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1 . INTRODUCTION 


The Large - Area Crop -Inventory Experiment (LACIE) .is -an inter- 
agency endeavor of the Rational Aeronautics -and Space Administra- 
tion (NASA), the National Oceanic and .Atmospheric. Administration 
(NOAA) , and the U.S. Department of Agriculture (USDA) . Its pur- 
poses are to test and demonstrate the accuracy and economic 
importance of utilizing remotely sensed data from the Land 
Satellite (Landsat) in conjunction with climatological and 
conventional- agriculture 'data- to -produce timely estimates of the 
production of a major world crop prior to harvest. 

The -LACIE Accuracy Assessment-' (AAj effort is designed to check 
the accuracy of LACIE products (estimates' of wheat production, 
area, and yield) throughout - the growing seas.on, to determine if 
the operational procedures are sufficient to satisfy LACIE 
project goals and objectives, and. to ’ identify problem areas in 
the estimation process. 

Most of the AA .studies , are .conducted in the U.S. Great Plains 

(USGP). region and Canada since these regions have the most-. 

ground-truth data available. However,, the AA studies in these 
- * . * & 1 

regions are ^also designed to promote the development of LACIE 
procedures which can be used to obtain accurate estimates for 
other parts of the world. I 

,1.1 ' PHASE III AA OBJECTIVES - i 

The following are the specific objectives for AA in Phase III. 

a. To make comparisons throughout the growing season between 
the LACIE and -the USDA Statistical Reporting Service (SRS) 
reference standard estimates of wheat \ production, area, 
and yield for the USGP region. 
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b. To determine whether the LAC IE production estimates are 
meeting the 90-90 project goal (to be within 10 percent of 
the true value with a confidence of 90 percent) . 

c. To conduct investigations of the error sources in the LACIE 
estimates and, where possible, to relate these error sources 

• J to causal elements in the LACIE estimation processes . 

i .. • 

d. To assess .the accuracy of the. LACIE .estimates for the U.S.S.R. 
to the extent permitted, by the -available data. This will 
include comparisons throughout the growing season between the 
LACIE estimates and. the .USDA Foreign Agricultural Service (FAS) 
estimates for the U.S.S.JR. . 

e. To investigate the accuracy of LACIE proportion estimates in 

the U.S. Great Plains and in selected regions of Canada, 
using blind site data. ■' ‘ 

This document was prepared by Lockheed Electronics Company, Inc. 
(LEC) , Systems* and Services Division,, Houston, Texas, under con-' 
tract NAS 9-15200 for the Earth Observations Division, Space and 
Life Sciences Directorate/ Lyndon B. Johnson Space Center (JSC) , 
of NASA. Inputs Were received from NASA, USDA, u and NOAA 
personnel. 
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2 . BACKGROUND 


In the LAC IE project, remote sensing technology • is used in con- 
junction with meteorological .and conventional agricultural -infor- 
mation to examine three global crop seasons, each of which is 
designated as a LAC IE phase. 

'.Phase I, which began in January 1-975, was devoted primarily to 

, identifying .and -estimating wheat acreage in the USGP states of ■ 
Colorado, Kansas, Minnesota, Montana * .Nebraska, North Dakota, 
Oklahoma, South -Dakota.,, and, Texas. Classification analyses, were 
conducted in other selected areas, and yield model development 
and yield feasibility determinations were made over selected 
regions in the United States. Data from the USDA/SRS S' were com- 
pared with LACIE estimates to determine .the accuracy. of LAC IE 
performance. - - ■ > 

' ■ " , * . » 

Phase I AA activities were initiated -in July 1975, and tes.ts for 
the accuracy of wheat acreage estimates were conducted using seg- 
ments for which ground- truth data were available. ^Initially, 
statistical tests and comparisons ..of LACIE estimates with ground- 
truth data were made using data from 27 intensive test sites 
(ITS's) i-n eight states? then, to test a greater number of 
acquisitions in a more concentrated area, ground observations 
of harvested small grains were obtained from 30 LACIE operational 
segments in North Dakota and Montana. The identity of these 
sites was withheld from the Classification and Mensuration Sub- 
system (CAMS) analysts so that they would process them as ordi- 
nary segments. For this reason, they were called blind sites. 
After the data from the blind sites were processed by CAMS, the 
AA Team (appendix A) compared the results of the various sampling 
and classification procedures used. Approximately 340 special 
classification runs were conducted to support Phase I AA 
activities . 

-g&CEDING PAGE BLANK NOT FILfL 
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In Phase II, which began in October 1975 and covered the 1975-76 
growing season, emphasis ‘was on the evaluation of' LAC IE proce- 
dures over the USGP region .and on -the development of AA 
methodology. • : • • 

LACIE acreage, yield, and production estimates for spring and 
winter- .wheat were compared with the corresponding USDA/SRS 
estima'tesY- This was done for the USGP region, for various sub- 
regions of the. USGP {see section 4. 2)-,’ and' for each state in the 
! USGP .. Detailed -error • source investigations were made by comparing 
LACIE proportion estimates with ground-truth proportions for 
over 150 blind- site's and 2 7 'ITS' s. Estimates of the production, 
coefficient of variation (CV) , and bias were -used to evaluate 
the 90-90. '.criterion at ‘the USGP level; and a sensitivity analysis 
was performed to determine the effect of various errors on the 
LACIE production estimate. In the foreign area, 10 ITS’s in 
Canada were- studied and evaluated. 

* \ , , * V 

|In Phase III, which began -in October 1976, the emphasis of AA 
continues to be. the', detailed evaluation of- LACIE estimates and 
procedures over the nine-state USGP "yardstick" 2 wheat region. 
Investigations .will also be carried- out in the : U.S.S.R. and 
Canada, as mentioned in section 1.1. 
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3 . SCOPE 

To accomplish its objectives, AA requires the multiagency collec- 
tion of various types of data to support the Phase III evalua- 

‘ * i , 1 *i ' • » 

tions . This includes : 

a. Aircraft photography over all U.S. blind sites and ITS's and 
Canadian test sites and ITS’s in order to prepare^ maps for 
land-use annotations by USDA Agricultural Stabilization and 
Conservation Service (ASCS) personnel. 

b. Land-use annotations from ground observations by USDA/ASCS 
personnel over all U.S. blind sites and ITS 1 s and Canadian 
test sites and ITS ’ s . 

c. Special ground observances of plant height and ground cover 
data every 18 days over 15 fields from each U.S. blind site. 

d. LACIE imagery, interpretation, and classification data over 
all U.S. blind sites and ITS's and Canadian test sites and 
ITS's. 

e. Monthly wheat production, area, and yield estimates over the 
USGP from LACIE and USDA/SRS. 

f. Monthly wheat production, area, and yield estimates over 
regions of the U.S.S.R. and Canada from LACIE and USDA/FAS. 

The following is the distribution of blind sites, test sites, 

and ITS ' s in the United States and Canada . 

a. 143 winter wheat blind sites in the USGP 

b. 68 spring wheat blind sites in the USGP 

c. 30 spring wheat test sites in Saskatchewan (Canada) 

d. 24 ITS's, in the USGP 

e. 10 ITS's in the Canadian spring wheat region 
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Specific details of the scope and data requirements for Phase III 
AA are presented in table 3-1. 

Listings and locations of ITS’s and distributions of blind sites 
by state are presented in appendix B. 
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4. GENERAL TECHNICAL APPROACH 

Three groups of activities are required to implement the LACIE 
Phase III AAjplan and satisfy its objectives: 

a. Planning and data acquisition 

b. Data analysis and evaluation 

c . Reporting 

A detailed flow diagram of the Phase III AA program and related 
LACIE operational activities is presented in figure 4-1. 

Specific descriptions of the AA tasks associated with each group 
of activities are presented in section 6 . 

4.! PLANNING AND DATA ACQUISITION 

The first group of Phase III AA activities consists of identi- 
fying AA data requirements and monitoring the acquisition of the 
data. Most of the data will be acquired by LACIE operations 
(NASA ) , the USDA, and NOAA; but coordination is required by AA 
personnel to ensure that the data, collected are adequate to 
support the AA program. 

Part of this monitoring activity is conducted by members of the 
AA Team (appendix A) . In particular, they select the blind sites 
in the United States and Canada and coordinate action for 
acquiring ground- truth data from the blind sites and ITS’s in the 
United States and Canada. 

The specific tasks to implement the above activity are identified 
, as tasks 1 through 8 in figure 4-1. Specific descriptions of 
•these tasks are provided in section 6.1. 


4-1 



1 


PHASE III ACCURACY ASSESSMENT PROGRAM 


m 


JtMAE PAGE 

SQOIt QUALITY 




■ DATA ACOtllSJllON 
PREPROCESSING 
AND TRANSMISSION 
SX5SY5HU tDAPTSl 
« vtELOCSTIuaTICN 
SUKrtltw IrESl 


1. PLANNING AND 
DATA ACQUISITION 


"OR»«» 

TOHOKBLK'Ol 

OPCANlMTiOMtlWWI 
Data acquisition 


C1N1M FORCLIUA 
TO LOO CAL AND 
(MVfflONMEMlAL 

issraamw KtfU 

YULPMUOLL5 


LACK OPERATIONS ^ 


CCEACnOP 

CFLERDAR 

MOOILS 

LAN3MT DATA 
ACQUISITION 10 5. 
OREATPEAINSFULL 
FAAMfOATA) 

^ 

1 A A TASKS T AN 0 L 
j (JfcTLICT atMQ 

0 AEOU»EAinCRAft 

i 

* P«OTO 
IN TtlLPPf TATUM 

M PREPARE JlClO 

, OVEREATS. 

MREEKLT CROP 

*yS^* 01 

I 

EtRVlTU 

| SAVE CLASSIFICATION PESULTS 1 j 
1 PHUM U^ AAO CANADUM RLtNOl 


sbs vxr hhcat tas rantisii aiica 

PFOOUCTlOl AREA MILAT PRODUCTION 

AND YIELD DATA APE* MID YIELD DATA 

* CUR KUII YEAR * CURRENT YE AC; 

PREDICT JONS Mil DICTIONS 

• HISTORICAL ©AT* - X ST OR ICM DATA 


PAT* PVOJESSIWC Of 
BUNO SITS DATA TO 
SUPPORT SEGMENT 


■ DATA PROCESSING Of 
ITS DATA TO SUPPORT 
3EDMEHT I ENEL AM> 


CAH3 CLASSIFICATION OL<X> 

EVAUJATlCM DAT* SATES 

■ IMAGERY “ ““ 

■ ESTIUATCSA.se lfrS 

■ AL DOT LAWlS 

■ CLASSIFICATION 
DOT LABELS 

• CLUTTXR DOT 

CAULS 


i 'TS CROWD TRUTH IKVCNTODY I 
ir CI ) St CM ENT AHO flAtL ]• 

LEVELS I 


ACM IB ACY IKESUCMT ANALYSE CARD CV» LUAHONS 


<«) ©EltftMMf If LACK PRODUCTION ESTIMATPS raR IBGP I*IIL 

NEW CR «■( MELTING THE LACK MINE STATE N-N AT HARVEST 
CMTRIOK COALS 

• CAIILY 6EAWB0-Se CftlTtmOM EVALUATIONS (flVE-StATE LCUtLl 
mu BE reported in the f ipiit second and third ■neniv 

AA hfPORTS . 

• NINE STATF J0-» CAltPRIClN E VALIJAtirWL KIUL PC RFPvlRTln - 
IN THE FOURTH IN It HIM AMD tIMAL PHASE II AA REPORT 


CONDUCT FIRST ORDER ERROR COW CL EViUUATCMS 
PRODUCTION IRFOft 


2. DATA ANALYSIS 
AND EVALUATION 


CACP CALENDAR E VALUATKM 
DATA YEUCCSACKOP 
CALI NO AR (YICDICTIONS 


I SAMPLIN'* iVALUATIW DATA J 
TOR TST7 BUYOUT! CLA£9 I 
flCATION AN© CFCUNO Tmnx 1 


CMRS 

• M 30-JCIICW ESTIMATES 

■ RELATED ACRE AGE AN 0 


(*) DATA COLLECTION PJUFAHAJICM AA DATA (ASK DEVELOPMENT 

• [WUCT DATA f hum LACK LP I rations AND IK 1 * DATA 

rnoccsstMc tasks y 

• SCLCCT ArnicrntAtc data sits to support Ctuluattons 

• CALCULATE 0« mOCRAM FOB CALCULATl DM THE STAtJStlcr, 
TO SUPPORT ACCURACY EVALUATIONS 


^?) Cff CCTOE CH*5CCBRICTI(W CY1 

SMALL CRAINS PROPORTION ISnw i 

(S) CAMS IV ALU Allow CODE 
(J?) ACQUISITION HISTORY 

(Sj FAQ (ificn 

CTOP CALI HOAR IRA dr 


COT LA set IMG 
@> ciuJTf ra’.G 
@ CLASSIFICATION AOCUHPC 


ITS EVALUATIONS 
INYtSTlCAIlDM CM Tl 


3. REPORTING 


PREPARE SPECIAL A* M*I|*G EUtn f CNlCFlFCS 
STATUS REPORT* AND PRESENTATIONS AS AEOL'IRCO 
TO SUPPORT LAClC WAHAGCUCUT REQUIRE HE NTS 




<«l) THROUGH @ (.* 


ISO PREPARE MONTHLY AA (YJiCK irm* REPOBIS r 
v DUfIJNCi PHASE HI THESE niPUHIS PTIIL COKEBT 
DF ifCHVlCAV COMMENTS OH BOTH 1HC US ,1 

and use r cauvs and car s cased air l{ 

COMPAQ ISO** of EACH ANO SYS OP f AS II 

PRQDLJCTILIN AREA ANO YlEtO LSIIHAItS L 
THE AA OVICR tUUA REPORTS MU BE itSUtOT 
? DAY'S AFTER RELEASE of THE A [I All D CMB 
EMI CAR REPORTS | 


» ffltrAnt AA INTCRI'E RIPORlS AND FINAL PMASI ill 
AA REPORT Thl fINAI AA HFPTTRT WILL DC 
DEVELOPED iHAOUUtl INI IIERAWt PROCESS Of 
REPORTING AliO RtVlIVnnC Hit INTERIM REPORTS 
THHOUCIIOUT Till QROITIMO SEASON IMF SCHEDULE 
FOB THESE REPORTS ARI 


• FOuRlH in ((SUV IMARCN IJ . FINAL A1 
PTIHEER ALD SFRU-C « lit At 

• FINAL PHASE III AA ntPORf I WAY W3I 


I - AJ HARVEST VTYTtR 
FINAL ATHARVISI 


Figure 4 1. Schematic diagram of LACIE Phase III Accuracy Assessment. 


4-2 


I 










4.2 DATA ANALYSIS AND EVALUATION 


The second group of Phase III AA activities involves the analysis 
of the data collected during the Phase III data acquisition 
activities. Various subsets of the Phase III data- are prepared 
to support the analyses performed by AA. ' These basic data sets 
are identified by the .numbers \ in 'the square symbols in figure 4-1. 
Similar 'numbers on the right side of the diagram indicate the 
specific analysis to which each data set is applied. 

The Phase -III AA evaluations are. carried out at the country level 
for the U.S.S.R. , at selected sites in ■ the ‘Province of Saskatche- 
wan in.-. Canada., and;.in -the nine-state USGP "yardstick! 1 region in 
the United States. Within this region ,- evaluations are usually 
made for each • individual state and for the -following smaller 
regions . • . ■ . 

a. The U.S.- southern Great Plains region (USSGP) — This region 
consists of Colorado, Kansas, Nebraska, Oklahoma, and Texas. 

. These states- have winter wheat -only and therefore could also 
be called the "winter -wheat states . " LAC IE estimates of 
wheat production are available /for :the USSGP from February 
-through October. . ... - 

b. The spring wheat states [ (SW states), Minnesota and North 
Dakota] — These states have spring wheat only. LACIE esti- 
mates of wheat' production- are' available from August through 
October. 

c. The mixed wheat states [(MW states)/ Montana and South 
Dakota] — These states have both spring and winter wheat. 
LACIE estimates of wheat production are available ‘ from 
August through October for spring wheat and from February - 
through October for winter wheat. 

d. The U.S. northern Great Plains region (USNGP) — This region 
consists of the two spring wheat states and the two mixed 
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e. The USGP region — This region consists of the nine states of 
the USSGP and the;USNGP. 

To determine the magnitude and components of error in LAC IE esti- 

' * - * * 

mates and .to ascertain whether or not the LACIE production esti- 
mates are satisfying the 90-90 criterion, AA personnel do the 
following: . . 

a. Determine the relative differences^ between LACIE and USDA/ 

SRS estimates of wheat production, area , s and yield over the 
various regions of the USGP "yardstick" region and between 
LACIE and- USDA/FAS estimates of wheat production, area, and 
yield in the U.S.S.R. ’In addition, ’ significance tests are 
made to compare these estimates. These data are reported 
throughout the Phase III growing season in the various AA 
quick-look reports, in the interim AA reports, and in the - 
final AA report. 

b. Determine if LACIE production estimates are meeting the -LACIE 
project 90-90 goal. ' These .evaluations are carried out at the 
country level for -the U.S.S.-R. and at the 'USGP level for the 
United States. In the United States, early season evaluations 
are .based on ,f ive-state or seven-state winter wheat data 
projected, to the nine-state level and -are. reported in the 
first and second AA interim reports. Nine-, state 90--90 cri- 
terion evaluations are made when the production estimate and 
standard error become available for the USGP. They are 

* i 

reported in the third and fourth interim reports. 

i 

c. Conduct detailed investigations of error sources within 
LACIE production estimates for the USGP, which shall consist 
of the following. 


1 


Relative difference = 


LACIE - SRS T 
LACIE 
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6 First-Order Error Source Investigations. These are _ 

studies of those errors contributing to. LACIE production 
estimates which can be estimated using. LACIE estimates, 
reference estimates (USDA/SRS) , and historical and blind 
site data. The effect of each error component is assessed 

. t , 

by estimating the error in LACIE production estimates 
caused by removing that error. The first-order error 
components to be 'evaluated are: 

a. Yield Error Source Estimation 

(1) Yield Bias 

(2) Yield Variance 

b. Area Error Source Estimation 

(1) Sampling Variance Estimation 

(2) Classification Variance Estimation 

(3) Classification Bias Estimation 

(a) Segment Level Bias Estimation 

i. CAMS Wheat Estimation Bias 
. ii. Ratioed Wheat Estimation Bias 

1. Small Grains Estimation Error 

2 . Wheat/Small Grains Ratioing Error 

(b) Country Level — Group III Ratio Error 

• Second-Order Error Source Investigations. These investi- 
gations examine in further detail the error components 
identified and/or quantified in the first-order error 

source evaluations . The second-order error source evalua- 

1 

tions are directed toward the investigations of problem 
areas that have been identified during Phase I and Phase II 
and toward the examination of error effects that are asso- 
ciated with the operational implementation of new procedures 
and equipment (e.g,. , Procedure 1 and the IMAGE 100) in the 
LACIE Phase III analytical process. A detailed decription 
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of the second-order • soutee investigations i's presented in 
section 6.2.5. ~-- r ' • 

d. Conduct special investigations of LACIE estimates over 

selected areas of the U.S.S.R. and Canada. These investiga- 

i * ■ * ( 

tions include: 

* , ' c * *- ; _ '* h * '* 

• Evaluation of the CAMS .proportion estimates for the -30 
Canadian test sites and 10 ITS 1 s by comparing them with 
ground-observed proportions for the same areas. The 
ground observations are arranged by the’ Canadian Centre 
for Remote Sensing. • • * 

The specific tasks for implementing the above activities are 

* j . - / * , j , , 

identified as tasks 10 through 39 in figure. 4.1. Specific 
descriptions of these tasks are provided in section 6.2. 

4.3 REPORTING 

Reporting for Phase T-II AA consists of -the following three types 
of reports. 

a. Special' AA 1 management briefings 

b. AA monthly quick- look reports 

c. AA interim and final reports 

4.3.1 SPECIAL AA MANAGEMENT - BRIEFINGS ' 

AA personnel will provide LACIE management with special briefings 
and presentations bn the status of AA data acquisitions and 
special problems during the 1916-11 winter- and spring-wheat 
growing seasons. These briefings provide timely responses to 
management requests for information about LACIE accuracy through- 
out the growing season. 
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4.3.2 AA MONTHLY QUICK-LOOK REPORTS 

These reports contain an evaluation by AA of the LAC IE estimates 
reported in the Crop Assessment Subsystem (CAS) Monthly Reports 
(CMR's) , the CAS Unscheduled Reports (CUR's), and the CAS Annual 
Report (CAR) . They are released 1 week following the release of 
a CMR or a CUR if the corresponding SRS or FAS estimates are 
available. Otherwise, they are released 1 week after the release 
of the SRS or FAS estimates. 

4.3.3 AA INTERIM AND FINAL REPORTS 

The interim reports are released in May, August, November, and 
February; and the final report will be released in April. These 
reports describe all the results obtained by AA up to the time 
that each report is written. 

The basic reporting formats and the suggested content of these 
reports are provided in the detailed task descriptions presented 
in section 6.3 of this plan. 


4-7 



5. '• SCHEDULE AND RESOURCE REQUIREMENTS 

The schedule and resource requirements for implementing LAC IE 
Phase III AA are presented in the following . sections . 

5.1 SCHEDULE * ..... 

The Phase III AA schedule is presented in figure 5-1. 

5.2 RESOURCE REQUIREMENTS 

The resource requirements for LACIE Phase III AA are shown in 
table 5-1, which summarizes the manpower and computer -require- 
ments associated with specific AA tasks or task groups. 

• CCEDJNG PAGE BLANK NOT F1LWL 


5-1 



TABLE 5-1. - PHASE III ACCURACY ASSESSMENT RESOURCE ESTIMATES 













































TASK/ACTIVITY 

REFERENCE £ CONTROL MILESTONES 

3 0 RESEARCH TEST AND EVALUATION 
31 ACCURACY ASSESSMENT 

311 PREPARE ACCURACY ASSESSMENT PLAN .. 

312 ACCURACY ASSESSMENT MONTHLY REPORTS 
31.21 USSR CAS MONTHLY REPORISfCMRL. . 

31.2 2 USSR QUICK LOOK AA REPORTS 
3.12 3 US CAS MONTHLY REPORTS (CMft) 

31.2 4 US QUICK LOOK* AA REPORTS .. , , 


313 ACCURACY ASSESSMENT DATA ACQUISITION AND CVAtUAl 

3131 A/C PHOTOGRAPHY . . 

3132 OROUND TRUTH PACKET PREPARATION 
31,3 3 OROUND TRUTH INVENTORIES 
313 4 15 FIELDS/B S. OROUND TRUTH ACQUISITION . . 

313 5 CLASSIFICATION DA1A ACQUISITION AND PREPARATlO 
3136 GROUND TRUTH/CAMS DATA PROCESSING |SF-12) 

313 7 ACCURACY ASSESSMENT DATA PROCESSING . 

3 U fl ACCURACY ASSESSMENT DATA BASE 

MAINTENANCE AND STATISTICAL. ANALYSIS 

3 14 ACCURACY ASSESSMENT INTERIM AND FINAL REPORTING 
3141 REPORT DEVELOPMENT/TECH PUBS PREPARATION. 

314 2 INTERIM ACCURACY ASSESSMENT REPORTS ' 

3143 FINAL ACCURACY ASSESSMENT REPORT 



0 /°- 


%'S 


* 


d* 


Figure 5-1.— Phase III AA schedule 
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6 . TASK DESCRIPTIONS 

Detailed descriptions of the- tasks that comprise AA for Phase III 
are included in the following subsections. 

6.1 PLANNING AND DATA ACQUISITION 

One of the. three major elements of Phase III. AA is the planning 
and coordination of- analytical and data acquisition activities of 
the program. This involves, (1) the determination of the basic 
data required to support the program, (2) the coordination and 
scheduling of data acquisition activities, and (3) the monitoring 
of data quality to assure that the data acquired are of satis- 
factory' technical quality. 

The AA .program depends upon the LAC IE functional elements [CAMS, 
CAS, Yield Estimation Subsystem (YES) , and Data Acquisition, 
Preprocessing, and Transmission Subsystem (DAPTS)] to provide a 
majority of the data necessary for AA evaluations. Specific 
task descriptions are not included in the AA plan for these LAC IE 
operations. Only those AA data acquisition tasks where direct 
involvement by the AA Team or by support personnel is required 
are described in the following paragraphs. 

6..1..1 PLAN .DEVELOPMENT. 

The initial activities of Phase I XI AA are directed to the devel- 
opment of an AA plan. This involves definition of the program 
scope, the data/resource requirements, .the schedules, and the task 
descriptions of the activities planned for Phase III. The proposed 
activities of the Phase III AA program are documented in this 
Phase III AA plan. 

6.1.2 DATA REQUIREMENTS 

Required inputs from the LACI-E functional > elements have been iden- 
tified through the planning for AA investigations. The basic 
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inputs are defined in the following subsections. As special prob- 
lems are identified', these requirements may be expanded to satisfy 

- , ,* , . f i 4 , , f - 

newly defined AA needs.’ 

6. 1.2.1 Classification and Mensuration Subsystem' .(CAMS) - 

CAMS personnel provide the" following : inf ormat-idh to’ 'the AA program 
for each 1976-77 analysis performed 'by CAMS and for - which results 
are passed- to 'CAS. This 5 information 1 is also supplied for each 
classification of' - each ITS. 

a. Segment number. 

' • * * * ' . , ’ . ’ i) i ■ . • . . ■ * * ' ‘ . 

b. Day of the year (DOY) , biostage, and. data quality indication 
for all acquisitions used by an analyst prior to classification. 

c. Dot labels and designated other /designated ' unidentifiable (DO/ 
DU) ‘‘field definitions prior* to -classification. 1 ■ 

d. DOY, biostage, and data quality indication for any acquisition 
used by an analyst after classification but prior to 
evaluation. 

i * , , . . ' * . ■ - / 

e. . Dot labels and. DO/DU field definitions after- classification, 

if different from (c) . < 

f. Unconditional cluster labels for each pixel as applicable [to 
be supplied via the Information Storage, Retrieval, and Refor- 
matting Subsystem (ISRRS-) ]•. 

g. Classifier label ’applied to each pixel (to' be supplied via 
ISRRS). The- 'bin value of wheat and other categories must be 
kept constant (e’.g.V wheat ■=- 239/ honwlieat = 143) . 

' * - ^ r ‘ i 1 ' * ' 

h. Prior probabilities used. 

i. Classification run number. 

j. DOY of the acquisitions used in classification. 

k. - CAMS evaluation '"bode - . 

l. Date of analysis. 
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m. Bias correction applied to classification result. 

n. Identification of dots used to label cluster's. 

6 . 1 . 2 . 2 Crop- Assessment Subsystem (CAS) 

The following data and information inputs are required from CAS 

during Phase III AA: 

a. U.S. and U.S.S.R. CMR's and any related CAS unscheduled reports 
(CUR's), which include LAC IE Phase III estimates of acreage, 
yield, and production as they are developed throughout the 
growing season. 

b. The standard statistics for area, yield, and production for 
LACIE Phase III U.S., U.S.S.R., and Canadian estimates, which 
include the standard deviation, the CV, the 90 -percent confi- 
dence limits, and the probability of less than TO percent 
relative error. 

c. A list of segments used in the aggregation, their estimated 
percentage of wheat, percentage of small grains, and acquisi- 
tion date. 

d. Wheat/ small grains ratios used by CAS in aggregations. 

e. Publication of the CAS data base at the end of the crop year 
(e.g., PC, GPC, etc.) 

6. 1.2. 3 Yield Estimation Subsystem (YES) 

The following data and information inputs are required from the 

YES* during the Phase III AA: 

a. Adjustable crop calendar (ACC) data used by CAMS to support 
analyst-interpreter classification activity over the blind 
sites, 24 U.S. ITS's, and' 10 Canadian ITS 1 s . 

b. Results from the Phase III, operational .yield .model for each 
zone in the USGP, for each zone in the U.S.S.R. where yield 
models exist, and for the years 1964-76 in a stepwise fashion. 
(This data requirement is similar to that for the Phase I and 
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Phase II yield feasibility studies.,) These results are 
required to support the first-order .yield investigations. 

6. 1.2. 4 Data Acquisition/ Preprocessing, and Transmission 
Subsystem (DAPTS) 

The following data and information inputs' are required from the 

DAPTS during Phase III AA. 

, , , * -v i 

a. The following ground-observation data for all U.S. blind 

sites: ’ ‘ ‘ 

• Aircraft photography (the basis for development of field 
overlays that are used in documenting inventories and 
interpreting 'signatures; see section 6. 1.4.2 for specific 
requirements) . 

• Completed fall early season winter wheat and at-harvest 
spring and winter wheat inventories of the blind sites 
conducted by USDA/ASCS personnel using the instructions 
and data recording forms presented in appendix D. 

• Eighteen-day observations of crop height and ground cover 
over 15 wheat fields in each blind site. 

b. The following ground-observation data (to be collected as a 
part of LACIE operations) for all U.S. ITS's: 

• Aircraft photography (the basis for development of field 
overlays that are used in documenting inventories) . 

i 

• Completed fall wall-to-wall and spring inventories of the 
ITS's conducted by USDA/ASCS personnel utilizing the data 
forms presented in appendix E. 

• Completed 18-day periodic observations of approximately 50 

fields from the ITS's throughout the wheat growing season 
until harvest, to be taken within 3 days of Landsat over- 
flights': (The USDA/ASCS - personnel make and record these 

observations on the forms presented in appendix E.) 
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c. Ground' observation of wheat growth-stage data {after each 
growth stage) acquired by USDA/ASCS personnel, .over' U.S. 

ITS ’ s . . , . . - . . . 

d. Copies -of mean historical crop calendars based on the last . 
15 years-,, if- available,, for each acreage stratum for the USGP 

.states and- Canada and copies of the -ACC's, as - available every 
2 ' weeks by" CRD . ■ ■ . * ’ ■ 

e. "Agriculture reports: 

® • U.SDA/SRS. .reports containing current information on wheat 
acreage, yield, and production for the .United States 
i at the state level. These. shall be made available to AA 
personnel on the. day of release. 

x < - * ’ ’ 

9 USDA/FAS reports containing current year information on 
wheat acreage, yield, and production for Canadian and 
U.S.S.R. wheat growing areas. These shall be made avail- 
able to AA personnel on the day.- of release. 

f. Historical agricultural statistics: ' - 

• USDA/SRS data on wheat acreage, yield, and production in 
the United States for 1970-75. 

© USDA agricultural census data-, for 1969 and--1974. 

g. Other required data sets as specified by AA personnel to sat- 
isfy special investigations that may, be requested, by LACIE 
project management. AA personnel will specify- any such 
requirements to DAPTS as soon as possible after. the data 
requirements are identified. 

6.1.3 S ITE SELECTION 

For Phase III, the AA team will select 143 winter. wheat and- 69 
spring .wheat -blind sites in the United States and 30 test sites 
in Canada. 
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6.1.3.X 


U.S. Blind Sites 


The U.S. blind sites were selected during the fall of 1976 so 
that aircraft photography could be obtained earlier than was 
the case in Phase II. The location and identity of all U.S. 
blind sites will remain unknown to the CAMS data analysts so 
that these sites can be processed as regular segments. The 143 
winter wheat blind sites were selected by a random draw strati- 
fied by CRD from five states' in the USSGP (Colorado, Kansas, 
Nebraska, Oklahoma, and Texas) and from two mixed wheat states 
(South Dakota and Montana) . The 69 spring wheat blind' sites 
were selected from the two mixed wheat states (South Dakota and 
Montana) and from the two spring wheat states (North Dakota and 
Minnesota) . The distribution of blind sites by state is included 
in appendix B (see table B-3) -. 

6. 1.3. 2 Canadian Spring Wheat Test Sites 

Thirty spring wheat test sites will be selected from the Province 
of Saskatchewan. These sites are similar to blind sites except 
that their identity is known to the analysts. 

6.1.4 PREPARATION OF BLIND SITE FIELD OVERLAYS 

Field overlays -will be prepared from aircraft photographs of the 
blind sites and will be used to record the land-use information 
obtained by observation on the ground. The following items are 
required to prepare the blind site field overlays. 

6. 1.4.1 Aircraft Maps 

After selection of the blind sites, Landsat imagery is used to 
determine the true position of each site. Analyst-interpreters 
determine these positions using production film converter (PFC) 
products, record the latitude and longitude to the nearest 
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0.1 minute, and plot the position of the segment on a 1:24 000- 
scale or 1:12 500-scale map. These maps are then used by air- 
crews in acquiring' the aerial imagery. 

6. 1.4. 2 Aircraft Photography 

Aerial photography is collected using color infrared film. If 
possible, this photography is obtained from a high altitude so 
that a single photograph covers the entire site. If this is 
not possible, the flight is made at an altitude of 6000 to 
7200 meter's (20 000 to 24 000 feet) and two flight lines are 
flown for each site with a 20-percent sidelap. Four frames are 
collected for each flight line- with a 30-percent forward overlap. 
All imagery must be collected no later than 4 weeks prior to. 
ground-truth collection. Predesighated flight lines are estab- 
lished for each blind site. 

After aerial imagery is acquired for the blind sites, each frame 
is checked to verify that the site was covered and that the 
imagery is of sufficient quality to be used by the USDA/ASCS 
personnel, in. collecting ground-truth data! 

6. 1.4. 3 Field Overlays and Field Segment' Kits 

If the imagery' is of satisfactory quality to be used by USDA/ASCS 
personnel, transparent overlays are prepared. The overlays are 
then placed ’ in field segment kits that are forwarded to USDA/ASCS 
personnel in the appropriate county for use in acquiring ground- 
truth data. These kits include: 

« A color infrared 2X or 4X print of the segment with boundaries 
on the field overlays . (The 4X enlargement is used for high- 

altitude photography.) 

• A topographical map of scale 1:250 000 showing the sample 
segment location and' boundaries. 


6-7 



9 Crop identification key with standard annotation for document- 
ing land use. 

-• Survey manual with a brief definition of field procedure's 
developed at JSC providing guidelines to USDA/ASCS personnel 
for recording ground observations of the LACIE blind sites 
(appendix. D).. 

6.1.5 BLIND SITE FIELD DATA ACQUISITION 

USDA/ASCS personnel provide complete inventory data based on 
ground observations. The data for each field are annotated on the 
overlay according to the standard crop symbols identified in the 
crop keys provided in the JSC instructions to -USDA/ASCS for making 
LACIE segment inventories (appendix D) . These inventory packages 
are to be completed by USDA/ASCS personnel and forwarded to JSC 
to be logged and tracked by DAPTS. 

All blind sites- -in the USSGP that have an early season planted 
inventory will have ,15 wheat fields chosen and annotated on the 
overlay by USDA/ASCS personnel: 5 below average stands, 5 aver- 

age stands, and, 5 above average stands. The USDA/ASCS personnel 
will identify the plant height and ground cover of each of these 
fields at this time.. Beginning on April 6> the USDA/ASCS per- 
sonnel will begin to revisit these 15 fields in concert with the 
Landsat overpasses so that classification performance can be 
related to wheat field stands . Also beginning April 6 , similar 
observations will commence over all blind sites which have been 
planted. 

As discussed in appendix H, software is being developed which 
will be run in the background' mode on the PDP 11/45 and will 
determine, for each of these fields, the amount classified as 
wheat and the amount classified as- other for each classification. 
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Prior to the time that this software is ready, manual interpre- 
tation is necessary. To do this, AA personnel will ask CAMS, to 
i pull the packets;. AA will pull the 18-day field observations and 

S ' I" » ’ * * 

the ground-truth overlay. The latest imagery used in classifi- 
cation and the classification map from the packet will be pro- 
| jected onto the ground truth overlay, and the percentage of each 
of the 15 fields classified as wheat will be estimated and - 
recorded. 

Fields 1 through 15, recorded -by USDA/ASCS personnel,, are assigned 
numbers ranging from 300 through 314 on the ground-truth overlay. 
,For those segments with training, dots defined on the appropriate 
PFC, each dot or small-field group of four dots will be .verified 
| in terms of the label given by the analyst versus the ground- 
: truth labei. Likewise, areas of other crops which are classified 
as wheat are estimated by AA personnel . When the automated system 
[becomes operational, these manual . interpretations will be’ used for 
!order-of-magnitude verification tests-. ' . ' 

6.1.6 PROCESSING BLIND SITE GROUND-TRUTH DATA TO SUPPORT SEGMENT- 
LEVEL AA INVESTIGATIONS - , - . . 

The early season blind site ground-observation data will be 
processed according to the procedures us.ed in Phase II. These 
procedures require the LACIE cartographic technician to plot the ; 
LAC IE segment boundary on a product 1, 2X photograph. Using the 
area mode ; feature of the H. Dell Foster digitizer, -the- technician 
planimeters or measures the segment area in thousandths of a 
square inch on, the photograph. Next, the proportions of wheat, 
small, grains,, abandoned wheat, and abandoned small grains (in 
each case separated into spring and winter) are -determined by - 
planimetering the area .for each of these classes and dividing by 
the total area in the segment., These proportions are used in 
various - investigations described in section 6.2.. 
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The late season blind site ground-observation data will be proc- 
essed in two ways to determine proportions. The first method is 
designed to obtain quick estimates of the proportions and the 
second (slower) method is designed to obtain the exact 
proportions . 

t 

The first method involves placing- a grid containing 400 dots over 
the ground-truth annotated aircraft imagery and determining the 
class of each dot. The proportions in the image are then assumed 
to be the same as the proportions in the dot samples. It is 
expected that the standard error in this procedure will be 
±2 .4 • percent. Proportions will be determined for wheat and 
small grains, and these classes will be further broken down into 
the categories of spring, winter, harvested, and abandoned. The 
following list gives the schedule for completing these estimates : 

a. North Dakota (27) August 25 

b. South Dakota (19) — August 31 

c. Oklahoma (20 — September 7 

d. Remaining USNGP (69) — Septembe^l9 

e. Colorado (13) — September '21 

f . Remaining USSGP (41) — September 26 

g. Canadian Sites (30) — O.ctober 1 

Here the number in parentheses is the number of segments. 

The second .method for determining ground-truth proportions 
involves using the Bendix 100 to determine the vertices of each 
field in. the segment. These will be stored on a magnetic tape 
along with the identification of the ground-truth class for each 
field. The proportions will then be calculated by the computer 
routine SPECTL, which is part of the pi-xel-level processing sys- 
tem described in appendix H. 
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6.1.7 FIELD DATA ACQUISITION IN INTENSIVE TEST SITES 

The field data acquisition from 24 U.S. and 10 Canadian ITS ' s is 
an integral part of LACIE operations. These sites are located 
prior to Phase III operations, and their identities and locations, 
are available to all LACIE personnel (see appendix B) . Field 
data acquired from these sites by USDA/ASCS personnel include the 
following: 

a. Aerial photography (once yearly) 

b. Field, maps annotated by USDA/ASCS personnel . 

c. Inventories of all fields (Figure E-l in appendix E provides 
an example of the ground-truth data reporting forms.): 

® After .fall planting for winter wheat areas . 

• "At harvest" for spring and winter wheat areas 

d. Periodic 18-day observations of a subsample (approximately 
50 fields) of each ITS coincident with each Landsat overpass 
(Figure E-2 in appendix E gives an example of the ground-truth 
periodic observation form used for recording these 18-day 
periodic observations.) 

These data are- forwarded’ to JSC to be processed, logged by DAPTS, 
and stored- in the LACIE Physical Data Library (LPDL) where this 
information is -then made available to AA personnel. 

6.1.8 PROCESSING ITS GROUND-TRUTH DATA TO SUPPORT SEGMENT- LEVEL 
INVESTIGATIONS 

Measurement of wheat and small grain proportions (both spring and 

winter) in the ITS will be done by using the Phase II procedure of 
• \_ 
adding field acreages from the inventory list. 
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6.2 ANALYSIS AND EVALUATION 

The .ana 1-ysis- -and- evaluation tasks described in this section are 
those presently planned to accomplish the Phase III AA objectives 
described in section 1.1. 


6.2.1 DATA COLLECTION AND PREPARATION 

| 

This task involves collecting the data required by AA from LACIE : 
operations and other sources. It is listed as a separate task 
because it involves considerable effort on the part of AA. The 
various data sets .required are shown in figure 4-1. . . 

6.2.2 COMPARISONS OF LACIE ESTIMATES WITH REFERENCE STANDARDS 
AS A METHOD- OF ASSESSING THEIR ACCURACY AND RELIABILITY 
DURING EARLY SEASON AND THROUGHOUT THE GROWING SEASON 

A prime concern of the LACIE AA program is to monitor and evalu- 
ate estimates made during early season and at regular intervals ' 
throughout the growing season. These evaluations are made 
through comparisons with the reference .standard (USDA/.SRS 
estimates) . ~ 


The statistic used for making these comparisons between the LACIE 
estimates of .wheat production, area, and .yield and the correspond- 
ing reference estimates. is the relative difference (RD) defined 
by : 


- - — wxr - DflRD »«• 


where LACIE stands for the LACIE estimate, of wheat production, 
area, or yield and STANDARD represents the corresponding refer- 
ence standard estimate. This definition expresses the difference 
between the two estimates- as a percentage of the LACIE estimate. 
In the United States these comparisons are made for each state in 
the USGP and for the various regions discussed in section 4.2. 

In the U.S.S.R., they will be made at the country level and 
possibly for certain regions. 
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Significance tests of no difference are made at the region or 
country level for the' LACIE production, area, and yield estimates 
for spr'ing’ wheat, winter wheat, and total wheat. In order to 
make a ' significance test, the LACIE estimate (of .wheat production, 

area, or yield) is assumed to be approximately normally distrib- 

•. 2 
uted with unknown mean y and variance . A test of the 

hypothesis 


H 


y = STANDARD 


versus the alternative 1 hypothesis 


H a : y / STANDARD 


is. then made using this assumption. The test statistic is given 
b Y . 

• ■„ ' LACIE - STANDARD 
L 

^ LAC IE 


which, under the null .hypothesis, is approximately normally dis- 
tributed with mean zero and variance one. The null hypothesis is 
rejected in favor of the alternative at the a-level of signifi- 
cance if 


Z > z 


a/2 


where z a /2 I s bb e (l “ §■) critical point of the standard normal 
distribution. For a - 0.10, z a / 2 ~ 1*645, and if |zj > 1.645, 
it is concluded that the mean of the LACIE estimator is signifi- 
cantly . different from. the reference standard estimate. 


These comparisons are designed to detect any abnormal divergences 
between the estimates and reference standards and thus to identify 
for further investigations of potential LACIE problem areas that 
might be associated with the divergences . 
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The results of these evaluations are reported in the Phase III AA 
monthly quick-look reports and in the interim and. final AA reports 
which are described in sections 6.3.2 and 6, .3. 3, respectively. 

6.2.3 DETERMINING IF LACIE PRODUCTION ESTIMATES MEET THE. 90-90 
CRITERION 

A 

Let P be the LACIE estimate of wheat production for the region or 
country, and let P be the ’true wheat production of the same region 
or country. The accuracy goal of -.the LACIE is a. 90-90 at-harvest 
criterion for wheat production, which is given by the following 
probability statement. 

• Pr [ |P - P| < 0.1P]* > .0.90 ■ (1) 

This states that the accuracy goal is for the LACIE at-harvest 
estimate of wheat production to be within 10 percent of the true 
wheat production with a probability of at least 0 ..9 . 


It is assumed that' the LACIE estimate, P, ’is normally distributed 

2 

with mean P + B and variance a^, where B' is the bias given by 

B = E(P) - P 


Under this assumption, equation (1) may be written as 


Pr 


-0.1 - 0.9 


B 


P + B 


, 0.1 t 1.1 


B 


CV(P) 


< z < 


P + B 


CV(P) 


> 0.90 


(2) 


P — (P -J- 

where Z = ~ — follows the standard normal distribution, 

aP 

N(0,1), and CV(P) is the coefficient of variation of P defined by 


CV(P) = 


CT P 

E (P )- 


rt^ 

. -p- 


P + B 


(3) 


The term 


B 


P + B 


is called the' relative bias of" P arid is given by 


B 

P + B 


E(P) ~ P 
E(P) 


(4) 
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It follows that the accuracy goal of LACIE is attained if 


0.1 - 1.1RB(P) 
CV (P ) - 


-0.1 - 0 . 9RB (P) 
CV(P) 


> 0.90 


(5) 


where $ represents the cumulative standard normal distribution. 
The enclosed region of figure 6-1 indicates combinations of 
CV (P) and relative bias for which equation (5) is satisfied. 


6.2. 3.1 Variance and Bias Estimation for the Wheat 
Production Estimate 

To apply the evaluation technique described in the previous 
section, knowledge of the variance, a~, and bias, B, of the 
LACIE wheat production estimate for a country or region is 
required. Since values of these parameters are unknown in LACIE, 
their estimates have to be obtained. The estimation of the 
production variance at different aggregated levels is described 
in detail in reference 1 and will not be discussed here. An 
estimate of the bias can be obtained by noting the difference 
between the LACIE production estimate and the corresponding 
US DA estimate of production. But this is possible only in the 
United States using USDA/SRS estimates. For foreign countries, 
USDA/FAS makes periodic forecasts which are generally for total 
grains production and are assessed using ad hoc methods. 

Although USDA/FAS estimates may be utilized to note any major 
problem, they cannot be used for a quantitative assessment of 
bias in a LACIE estimate. 

6. 2. 3. 2 At-Harvest 90-90 Criterion Evaluation 

* 2 A 

Given the estimator for <?£, an estimate of CV(P) is 

CV (P) = a|/P 
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variation (CV) of 




2 

where the computation of cx.g is described xn detail in reference 1 
and is provided by the CAS aggregation software. An estimator 
of RB (P) _ is 

A .A, A .A 

RB (P) = B/P 

where B is the difference between the LACIE production estimate 
and the corresponding US DA/S RS' estimate. 


The observed value of the left side of equation (5) , above, with 
CV (P) and RB (P) replaced by their estimates, CV(P) and RB(P), ' 
respectively, is subject to certain variability, which is intrac- 
table due to problems in obtaining a joint distribution of 
CV (P) and RB(P). However, if CV(P) is greater than 0.061, there 
is an indication that the LACIE estimator does not satisfy the 
90-90 criterion even if P is unbiased. Since CV(P) has been 
found to be very stable at the country, level (USGP level in the 
case of the United States) and less than 0.061, CV(P) is treated 
as the parameter CV(P) , and equation (5) can be solved to 
determine the tolerable values of RB(P) that would meet the 90-90 
accuracy goal. That is, given CV(P) , there exist real numbers 
R q (R o > 0) and R 1 (R^ > 0) so that equation (5) is satisfied for 

R q < RB(P) < R x 


or, equivalently. 


B < B < B 
o 


where B q = R q P/( 1 - R q ) and B^ = R^P/(1 - R^) , P being the true 
production. 


Suppose next that the LACIE production estimator is a 90-90 
estimator; i.e. , suppose CV(P) = CV{P) < 0.061 and RB(P) € [R q ,R^3. 
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Under this assumption , calculate the significance level 


SL = A max {min [Prob [RB (P) < rb(P)], Prob[RB(P) > rb(P)]]} 

RB{P)€ [R q/ R 1 3 

= max {min [Prob [B < b] , Prob [B > b] ] } (6) 

B<a V B i r ' 

A A A A 

where rb(P) and b are the observed RB(P) and B, respectively , and 
the two probabilities are computed assuming B is normally distrib- 

* r 2 , , *A 

uted with mean B and variance Since P is assumed normally 

, * 2 
distributed with mean (P + B) and variance a~, 

* \ ' Jr 



is also normally distributed with mean B and variance a~, provided 
P = The estimated significance level, SL, is an estimate 

of the probability of encountering the" observed difference, given 
that the LACIE production estimator is a 90-90 estimator. If 
SL is' small, say less than 6.10,' it is concluded .that the 90-90 
accuracy goal was not attained due to a consistent bias that is 
larger than' the tolerable amount. if SL is larger than 0.10, 
it is not immediately inferred that the LACIE production 
estimator is a 90-90 estimator. This is due to the fact that 
only one observation has been obtained to estimate B. However, 
results obtained from blind site analyses and other AA tasks are 
then considered for further assessment of whether or not the 
90-90 accuracy goal is achievable. 

6. 2. 3.3 Early Season 90-90 Criterion Evaluation 

Although the "official ,r evaluation of the 90-90 criterion is 
based on at-harvest estimates, it is of interest to evaluate how 
well LACIE is performing throughout the season. When data are 
available for both spring and winter wheat (generally after July) , 
the evaluation is performed in the same way as for the at-harvest 
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estimate. In order to gauge how well LACIE is performing early 

in the season when only winter wheat data are' available , a' 

method was- developed' -to project the -winter wheat results, for the 
» 1 
5- or 7 -state level to the- 9 -state total harves table wheat level. 


The 5- or 7-state relative difference between the LACIE and USDA 
estimates* is- taken as an estimate of the relative bias. The coef- 
ficient of variation, however,, is "projected" to the . 9-state level 
by ' 


cv 1 



(7) 


/N /\ ^ 

where ChA, is the current month CV(P) for* the 5- or 7-state wxnter 

K. 

wheat production estimate, and N_ and N TTC? are the corresponding 
numbers of allocated segments for the 5- or 7-state region and 
the USGP region, respectively. After the relative bias and 
coefficient of variation have been estimated, inference as to 
whether the 90-90 criterion has been supported is made using 1 
the evaluation procedure discussed in the previous section. 


6.2.4 ERROR SOURCE EVALUATIONS - FIRST-ORDER ERROR 
SOURCE INVESTIGATIONS 

A major purpose of AA is to attempt to ascertain the nature and 
characteristics of the error In LACIE Phase III production 
estimates. This' requires indepth investigations in which the 
error in LACIE production estimates is quantitatively and/or 
qualitatively associated with, various causative factors. However, 
■the error in production -depends on its sources .in a complex way; 
thus, it is 1 unrealistic to assume that the total' error component 
can be written as a sum of uncorrelated randorm components. Instead, 
the effects of the major components are evaluated by estimating 
the reduction in the prediction error _ of production, which 
results from eliminating that component of error. These major 
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TABLE 6-1.— PHASE IX CV'S AND RELATIVE DIFFERENCES 


Date 

Area, 

state 

Phase II CV 
for production 

Phase II 
relative 
difference 

Feb. 

5 

, 11 

-4.9 

Mar ; 2 5 

5 

10 

• -9.9 

Apr. 8 

5 

8 

-8.5 

May 7 

5 

8 

-1.6 

June 8 

5 

* 7 

11.4 


7 

8 

1.7 

June 29 

5 • 

7 

12.7 


7 

7 

4.7 

July 9 

5 

7 

-3.7 


7 

7 

-7.9 

Aug. 11 

5 . . 

.7, , 

-4.2 


7 

■ 7 

-5.6 


9 

6 

-14.7 

Sept . 9 

5 

7 

-6.6 


7 

7 

-6.6 


9 

5 

-13.6 

Oct . 8 

5 

7 

-6.6 


7 

7 

-6.5 


9 

5 

-13.8 

Dec. 17 

5 

7 - 

-7.2 

(final) 

7 

7 

-7.2 


9- 

• 5. 

-12.3 












components of production error are called first-order errors. 

They are graphically depicted in figure. 6-2. These first-order 
errors are the errors contributing to the LACIE production 
estimate which can be quantitatively estimated from LACIE esti- 
mates, reference standard estimates (USDA/SRS) , and historical 
(county census) and blind site data. Methods for evaluating 
these first-order error components are discussed in sec- 
tions 6 . 2 . 4 . 1 through • 6 . 2 . 4 . 5 . Section 6 . 2 . 4 . 6 ‘ describes the 
method for determining the effect of errors in yield, acreage, 
sampling, and classification on the production estimate. , 

6 . 2 . 4 . 1 Proportion Estimation Errors 

Proportion estimation errors are determined for the CAMS estimates 
of small grains and ratioed wheat.' If CAMS estimates wheat 
directly (i.e. , without using- ratio)- in Phase III, the proportion 
estimation errors for these estimates- will ' also be studied. 


The proportion error for a -given blind site ’segment is X - X, 
where X is the CAMS proportion estimate -and X is the ground-truth 
proportion. These errors- are. studied by plotting them as a 
function of X for each state in ! the USGP and for the USGP region. 
Also, statistics for these regions are calculated as follows. 


Let N be the number of segments allocated to a region (state or 
higher level) , and let n be the number of blind sites selected 

* /S 

randomly from these N segments. For a region, let X^ represent 
the CAMS estimate of the proportion of wheat in the i th segment 
and let X^ represent the ground-truth proportion of wheat in the 
\ith segment, where i = 1,*»*,N. Then" the average error y D is 
given by 


N 



i=l 
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Figure 6-2 


LACIE first-order error components. 


















( 9 ) 


The estimate of p D is given by 


■ n 



i=l,' ■ 

where the summation is taken over the n blind sites. 


Letting = x^ - X^, we may estimate the variance of D by 



( 10 ) 


Lower and upper confidence limits for the population average 

difference D are given by 
P 


u D t D ” t l-a/2 S D ; 

L 


— D + t. /0 S— 
Dy l-a/2 D . 


( 11 ) 


where ^- s va i ue of 1 - d/2 percentage point, from the 

Student's t distribution with (n - 1) degrees of freedom, corre- 
sponding to the desired- confidence level of 1 - a. 


The null hypothesis U D = 0 (i.e., no bias) is rejected at the 
a-level of significance if |d/S-| > t 1 -a/2' or e< 3 uivalentl y ' if 
the confidence interval given by equation (7) does not contain 
zero. • • 

The quantities D, S?r, , and y n are tabulated at the state 

and the USGP levels , arid the test is- used to determine whether 
or not there is bias at these levels. 
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6. 2. 4. 2 Contribution of the Classification and. Ratioing Errors 

to the Ratioed Wheat Proportion Estimation Errors at the 
Segment Level - . , 

This section describes the method used to study the contribution 
of the classification and ratioing errors, to the ratioed wheat 
proportion errors, which consist of the proportion bias and the 
proportion mean-square error. (MSE) .. , . 


A A t i 

Let r^ and x^, l = l,2,**.,n, be the estimates of r^ and x^ 
respectively, for the i th blind site, where r^ is the ground- 
observed ratio of wheat to small grains proportion; x^ is the 
ground-observed small grains proportion; and n is the number of 
blind sites. In this section, r^ is the forecast ratio of wheat 
to small grain's' proportions and is the CAMS estimate of the 
small grains proportion. 


The bias (B) and the MSE of the wheat proportion estimate 
obtained after ratioing may be estimated by. 


and 


n 


B = — ^ ^r.x. - r.x":) 

n / j \ i i ii ] 


i=l 


( 12 ) 


M ® = K L (% ■ r i x i) 2 U3) 

i=l 

It is clear that these 'errors are both caused by two factors: 
the CAMS classification of .small grains and the estimated ratio 
of wheat to small grains . The contribution of a particular 
error factor may be assessed by the reduction in the bias or MSE 
which would be achieved if that error factor were omitted. 
Specifically, the following formulas are used in this study. 
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a. Bias estimate with no ratioing error: 

n 


=•-’ ?X"( r A - r i x t) 


i=l 

b. Bias estimate with no classification error: 

n 


B 


* * n E (Vi..: Vi ) 1 . 


i=l 

c.. .MSE estimate with no ratioing error : 

h 


MSE 


• = - V* /r .x. - r.x. ) 2 
n JLj \ x x x x/ 


i=l 


( 14 ) 


(15)' 


( 16 ) 


d. MSE estimate with no classification error: 


MSE" 


n 


I E ( J i x i - r -i x i) 


i=l 


(17) 


These quantities will be calculated at the state and USGP levels 
and a sensitivity analysis will be conducted to measure the 
effect of classification and ratio error on the bias and MSE 
for ratioed wheat proportion. 


Data required for this investigation are: 

a. CAS ratios for every USGP blind site used in the aggregation 

b. CAMS estimates of spring and winter wheat and either a 
small grains or a spring or winter small grains estimate 
for every 1977 blind site. 

c. Ground-truth proportions for spring wheat, winter wheat, 
small grains winter small grains .and spring small grains 
for every 1977 USGP blind .site. 
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a large area in the United 


6. 2. 4. 3 Classification Bias 


The LACIE estimate' of wheat acreage; for 
States can be written *' : 


i 



W.X. 
X X 


( 18 ) 


A ' * 4 A 

where A is the estimated wheat acreage of the region; is the 

wheat proportion estimate in the i th LACIE segment, n is the 

number of processed LACIE' segments; and the W. are weights based 

1 1 

on historical and cartographic data. 


A , 

Corresponding to A is the true acreage. A, which can be written 

-n . 

a = 2 "r’i (is) 

i=l 

v * 

where is the true wheat acreage • for the county containing the 
i th segment and W* is the value of the weight which would give 
perfect Group III estimates of wheat acreage for unsampled 
counties. 


We can now write 

- c i + <x A - c ± ) + (X. - XjL ) 

= c i + ^i + £ i ( 2 °) 

where X^ is the true wheat proportion of the i th segment, 6^ is 
the sampling error, and e- is the classification error. Since 

* , . *^* » * * j ■ i 

the sampling was performed in an unbiased manner, we assume 


1 


The precise definition of'W^ depends on whether the 1th segment 
is used as part of a Group ill* estimate. 
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E{6^) = 0; however, we do not assume unbiased classification. 
Instead, let 0 be an average segment bias; i.e., 

E (e i ) =• 0 • (21) 

A A 

The bias in A is defined by E (A - A) , which is given by 


B = E(A - A) 



t- 

W.E (c . + 6 . + 
l i l 


t n 

i=l 

n 


n 

i=l 

2 

i=l 

< W i T Wf)c. + 

9 2 

i=l 

W. 

l 


( 22 ) 


Note that the first term of equation (22) represents a bias 

caused by the failure of the Group III ratios to be exact (i.e., 

* 

jP W^) ; whereas the second term is the average segment bias 
multiplied by the sum of the W^.' 

i 

At present, only the second term of equation (22) is estimated 
since good county- level data are not available for estimating the 
first term. 


The second term is estimated by (1) breaking up the large area 
into strata (not necessarily connected) for which equation (21) 
holds; i.e., the bias is constant, (2) estimating 6 by 


0 




the average proportion error on' a segment 


level in the k th' stratum where n, is the number of blind sites 

k A . 

in the k th stratum, and (3) aggregating 0 over the strata. 
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If B represents the estimate- of ‘ bias 'for^the region resulting 
from classification; a 90-percent confidence -interval 'for B 
(the true bias) can be constructed by 

(B - 1.645a, B -i- 1.645a) 

where a is an estimate of the standard error of B. 


If we assume Var(e.) = o* (a constant) -within the kth stratum, 

1 - CK n. ys ■' *2 

(x 4 - x. - 0) 

— and Var(Bl can 


2 

f ck 


then can be estimated by 2^ — — 


k 

c 

i=l 


n. - 1 1 
:k 


n. 


be estimated by Var(B) = ? $ ck( W ki ) , where is the weight 

k \i = l / 
for - the i th segment in the kth ‘stratum. 


6. 2. 4. 4 Estimation of the .Within-County Acreage Variances 
Resulting From Classification and Sampling Errors 

’ . . . * * * ' - * 

In order to estimate the within-county acreage .variances result- 
ing from sampling and classification errors, one first obtains 
Jthree basic regression models; namely, true segment proportion 
versus historical county proportion , LACIE segment proportion . 
versus ground-truth segment proportion, and LACIE segment propor- 
tion versus historical county proportion. Then these regression 

. 2 2 2 
.equations are used to obtain the estimates for a + a TT , a , and 

*222 222 2 sH'c 

a s + a c w ^ ei : e \ ° s r a c '- - a H represent, respectively, the 

contribution resulting from sampling, the contribution- resulting 
from classification, and the variance of the residuals resulting . 
from the regression of the current county proportion onto the 
historical county proportion. .(Assuming that is much smaller 
than a g , a H can be ignored in the calculation.) Finally, -the 
maximum likelihood estimation technique,, assuming normality, is 
used to obtain the optimal -estiinates. for sampling .and classifica- 
tion variances. The detailed description of this method is pre- 
tsented in appendix C , 
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6. 2. 4. 5 Yield Bias and Variance 


To support the evaluation of the 90-90 criterion for production, 
the yield estimates must be tested for bias and the accuracy of 
the corresponding estimated variances determined. 'In order to 
do this, the bias and variance are tested, as first-order error 
Source evaluations, for each zone for all truncations for the 
U.S.S.R. and the USGP using 10 or more years of independent test 
data. This is being done as a test and evaluation tasks The 
data to be used are as follows. 

a. , United States 1965-76 

b. U.S.S.R. 1958-72 

c. Canada 1967-76 


6. 2. 4. 6 Production Bias 

The production bias at the state level -is ■ given by 


Bt, = 

P. 

l 


E (P . - P. ) 
x x 


- E(P i )^P i 

= eia^.) - a ± Y. 


(23) 


where P^, A^, and Y^ are the true values of the production, acre- 
age, and yield, respectively, for the ith state in question, and 
i?i' A^, and Y^ are the corresponding estimates for these quanti- 
ties. Assuming A^ and Y^ are independent, one obtains 


B_ = E (A. ) E (Y . ) - A . Y .. 
p i ,i x xx 

A f A 

If one further assumes that Y. is unbiased, then E(Y.) 

x x 


(24) 


Y. , and 
l 


b p. - V E( V - V- 

x 

= Y.B* 

X A. 


(25) 
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where B A is the acreage bias for this 1th state. The quantities 

Y. and B are unknown, but an estimate, can be. obtained by 

;using the 1 estimates for Y. and B . Thus, 

r - • ■ 


A A A 

B n Y . B _ . 

P i lA i 


(26) 


The variance of B p : is given .’by 


Var I 


(v) - *± ™(\) 

and estimated by 


+ E‘ 


! (§ a j Var (Y ± ) ' '+ Var^ A< ^ Var(Y ± ) (27) 


/ y\ 

Var /B p 


i) = ^ vSr (X) 


A A 


A / A 


+ B~ Var (Y ± ) - Var 
x ± 


1&I 


A A 


Var(Y ± ) (28) 


For the nine-state level, the ■''production bias estimate B p is 


£ 


§ P. : = £ «i S A, < 29 > 


and the estimate of its variance is 


i 

A /A 


is ^ vir (\) 


The relative 


bias of the production estimate R(B p ) is estimated by expressing 
the production bias as a percentage of the LACIE production esti- 

J ' . i * * 4 ' * . * 

mate; that is, by ■ ’ ' * ' 


Vy.B 

R (B ) ‘ = — x ’100%' ' ’ 

P 


£Vi 


(30) 


6. 2. 4. 7 Effects of Errors in'Acreage, Yield, Sampling, and 
, Classification on. the .Production, .Variance 

The production variance consists of two major error components: 
acreage and yield. The acreage error 'may, be further subdivided 
into sampling and classification errors.. The effect of a par- 
ticular error is determined by the ‘ reduction in the production 
variance estimate when the error is omitted from the calculation 
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of that estimate. If there is only one yield stratum in a zone 
(state), the production variance is calculated. at the zone level 
and aggregated to higher levels . If a zone contains more than 
one yield stratum, it is subdivided into pseudozones, which are 
the intersections of the zone with the various yield strata. 

The production variance estimate is then calculated at the 
pseudozone level and aggregated to the zone and higher levels. 


Suppose the zone consists of H pseudozones, G- ,G~ , • • • ,G , with 

X £ H 

acreage estimates A z i ' A z2 ' * * * ' A ZH and predictions 

Y Z1' Y Z2' * * * ,Y ZH' res Pectively . Then the estimate of the produc- 
tion variance at the zone level is given by the following equa- 
tion, which also appears in appendix B of the CAS Requirement 
Document. 


H 


s: = 


(vLyL + - v^u 2 


(. Zi Zi 


Zi Zi 


Zi Zi 


i) .. 


i=l 


+ 2 X) Iti Y Zi Y ZH ( .2 7) 

i % »=i \>^ G i 3 1 


(31) 


where 

ii- 

2 

V . = 
Zi 


¥ = 
3k 


the estimate of the yield variance for the i th pseudozone 

the area variance estimate for the i th pseudozone 

the estimated covariance between A. in G. and A, in G„ 

3 i k it 


In order to determine the production variance without a given 

error term, equation (31) must be rederived with that term 

2 2 2' 2 

omitted. Let S ZA , S ZY , S z g, and S Z( _, be the state production 
variances without acreage, yield, sampling, and classification 
errors, respectively. One obtains the following expressions for 
these quantities. 
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(32) 


= E ( u 1i4l - v zi u z±) 


S ZY = E ( V Zi Y Zi " V Zi°Zi) ; 


H i-1. 


-e 

. i-2£«l. . V eG i 


^ 

<TG. k€G„ 

.1 £ j 


S ZS - E [ (1 - &)V Zi Y Zi + U Zi A Zi - (1 - 5)V Zi D Zi] 


H i-1 


2 EE Y Z iW (i - e >5 2 v) 

i=2 £=1 V ? € . G i keG £ / 
s zc - E ( ev zi Y zi + D zi A zi : ~'KAi) 


H i-1 . 


■EEw.fSS',) 

x =2 £=1 • ■ - \ J l £ / 


where £ is defined on page C-7 of, this report. 

Let S rA' S rY^ S rS' and S rC he the ra 9> onal_level Production 
variance estimates without acreage, yield, sampling, and classi- 
fication errors, respectively. These estimates can be obtained 
from the following expressions. 


■ .R . 


R. R 


* E s L + E E 


’rZZ • 


Z=1 Z'=l 
Z^Z' 
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( 37 ) 


R 



Z=1 



(38) 


(39) 


Here R is the total number 
if . Z th and Z'th zones have 


of zones in the region and S rZ2 , = 0 
no yield strata in common. Otherwise, 


rzz 


K=1 


A rzAz ' K U rK 


(40) 


where . , • 

A = the area estimate for the pseudozone corresponding to 

Xal\ 

yield stratum K in zone Z of region r 

2 

U rK = the squared prediction error for the K th yield stratum 
common to zones Z and Z 1 

C - the number of yield strata common. to the Zt/i and Z 1 th 
zones 


The estimates of the corresponding variances for a country are 
obtained by adding the corresponding estimates for all the 
regions in the country. These computations as.sume that the 
regional production estimates are uncorrelated. 


6,2.5 SECOND-ORDER ERROR SOURCE INVESTIGATIONS .. 

Second-order error source investigations are designed to study 
the dependence of the errors in the LAC IE system on various 
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causal factors. It is hoped that such studies will suggest 
changes in LACIE procedures that will lead to improved estimates. 

The second-order error source investigations have been grouped 
into two categories : segment level and pixel level . Segment 

level investigations start from data relating to a >7hole segment, 
such as a segment wheat proportion. Pixel level investigations 
start from data relating to pixels, or collections of pixels, 
such as dots or clusters. 

6. 2. 5.1 SegmentrLevel Error Source Evaluations 

The following paragraphs describe the segment-level data error 
source evaluation tasks that are planned during Phase III. 

6. 2. 5. 1.1 Effect of Bias Correction on Wheat Proportions 

The purpose of this task is to determine the error in the LACIE 
Wheat Proportion Estimate (X) before and after bias correction 
has been applied. 

To accomplish this, classification and ground-truth data (X) 
over the blind sites of the USGP are utilized. Comparisons of 
X versus X are made as well as X corrected (bias correction 
applied)" versus X. Aggregations are made such that the classi- 
fication biases and variances can be examined at the state, sub- 
region, and USGP nine-state levels with or without bias 
correction. 

6. 2*5. 1.2 Effect of Bias Correction on Small Grains Proportion 
Estimates 

This task is similar to that described in section 6. 2. 5. 1.1 
except that small grains estimates are evaluated and no aggrega- 
tions will be performed. - 
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6 . 2 . 5 . 1 . 3 CAMS Evaluation Code 

This task consists of the investigation of the relationship of 

< * J ) 

the CAMS code (describing the quality of the classification) to 
the proportion error (X - X) observed over the U.S. blind sites 
(211) . Comparison between the proportion error associated with 
the categories of CAMS evaluation codes is made and evaluated 
at state and region levels . 

6. 2. 5. 1.4 Acquisition History 

The purpose -of- these evaluations is -to see if the observed pro- 

A 

portion errors (X - X) depend on the acquisitions (as represented 
by the Robertson biostage) during the winter wheat or spring 
wheat growing seasons. The relationship of error magnitude to 
acquisition pattern is examined. 

The data set for this task includes CAMS classification data and 
ground-truth data during the 1976-77 growing season over the 
USGP blind sites recorded by biowindows. 

6. 2. 5. 1.5 Haze Effects Evaluations 

During Phase III haze effects upon classification are investi- 
gated through the acquisition of optical depth measurements with 
manual radiometers at each ITS at 10-minute intervals from 
30 minutes before to 30 minutes after each Landsat 2 overpass 
of the site. These data shall be collected from February (or 
end of dormancy) to the end of harvest. The solar radiometer • 
must be calibrated before and after field use and the data must 
be analyzed with respect to optical depth by November 1, 1977.- 

In addition -to the haze effects study of the above tasjc,. during 
Phase III, three auto tracking solar radiometers are placed in 
the three supersites (Finney County, Kansas; Hand County, South 
Dakota; and Williams County, North Dakota) by the field 'measure- 
ments team for use during selected overpasses of Landsat 2 . 
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Optical depth is used and calculated for six channels at rates 
greater than 1 per second for at least 30 minutes before till 
30 minutes after the Landsat 2 overpass. Wind speed and direc- 
tion, temperature, and relative humidity are also monitored at 
the same rate over the same period. 

Accuracy evaluations of the manually operated solar radiometer 
involve calculating the mean and standard deviation of the optical 
depth for each of six channels for the 1-hour’ observation period, 
/that is icorrelated with the Landsat. overpass , ■ over the 24 USGP 
ITS's. • The correlation .between, .these . data and labeling and 
classification omission and commission’ errors will.be determined. 

^ it ' , * 

6. 2. 5. 1.6 Crop Calendar Error Determination 

A major reference utilized by analyst-interpreters in their 
classification" procedures is' the nominal (meari historical) crop 
calendar and the adjustable crop calendar (ACC) . Since the 
LACIE ACC provides the latest reference information on the 
stage of development of wheat in an area being classified and 
estimated, it is necessary to determine the accuracy of this 
reference • information. This task is designed to determine the 
accuracy of the ACC . estimates of the wheat growth stage through- 
out the growing season <. ; - .. • 

The basic data set for these evaluations is the growth-stage 
data acquired by USDA/ASCS personnel over the 24 U.S. ITS's. 

These growth-stage data are acquired utilizing the ground-truth 
periodic observation form : presented in appendix E, figure E--2. 

In addition, growth-stage information is acquired over the 
following Crop Reporting Districts (CRD ''s) : 

a. Texas (IN, IS, 2N, 2S, 5N, 5S," 8N, 'and 8S) 

b. Kansas (all CRD's) 
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c. North Dakota (all CRD's) 

d. Montana (1, 2, and 3.) 

The CRD growth-stage data are acquired by' USDA/ASCS personnel 
using the growth- stage reporting forms presented in appendix F. 

t * 

The USDA/ASCS delivers these data to the DAPTS at JSC 30 days 
after completion of each applicable growth stage in each CRD 
stratum or equivalent. 

Plots will be made of the ACC outputs (for the ITS's), the mean 
of the ground observations of wheat growth stages, and the 
nominal crop calendar. Confidence estimates will be >made based 
on the distribution of the ITS ground-truth observations, and it 
will be determined if the ACC results fall within these limits . 
The relationship of the crop calendar information to known 
episodic events of the current year, such as drought, is inves- 
tigated and reported throughout Phase III AA, along with the 
assessment of the accuracy of the ACC. 

6. 2. 5. 2 Pixel-Level .Error Source Investigations on Blind Sites 

These investigations will provide a method of more accurately 
assessing the labeling, clustering, and classification perform- 
ance during Phase III. They are based on a new procedure for 
processing the ground- truth data which is -described in appen-- 
dix H. In this procedure, a "ground- truth tape" is produced in 
which the ground-truth data are presented as an image, analogous 
to the Landsat imagery and to the cluster and classification 
maps produced by CAMS. Each subclass in the ground-truth data 
has its own assigned grey-scale level on the ground-truth tape. 
The subclasses used are shown in table 6-2. In addition, certain 
individual fields (the 15 fields described in section 6.1.5 for 
blind sites and the 50 fields described in section 6.1.7 for 


6-37 



TABLE 


6-2. ~ SUBCLASSES USED IN AA INVESTIGATIONS AND THE CORRE 
SPONDING GREY-SCALE LEVELS ON THE GROUND-TRUTH TAPE. ’ 


• Subclass . - , • . 

r , . i 

• Grey-scale 
level 

Fields 1 to 80 

1 to 80 

Alfalfa 

90 

Beans 

91 

Corn 

92 

Safflower 

93 

Sunflower 

94 

Sudan grass • . 

> - 95 

Sorghum 

.96 

Soybeans,. 

97 

Sugar beets 

98 

Winter wheat 

99 

Spring wheat 

100 

Barley 

101 

Rye 

102 

Flax 

103 

Oats • • 

104 

Grass 

105 

Hay 

106 ' . 

Pasture 

107 

Trees 

108 

Same as '90-108 except: 

* 

Harvested 

115-133 

. Abandoned 

140-158 

Strip fallow 

165-183 

Strip tallow harvested 

190-208 

Strip,, fallow abandoned 

215-233 

Water 

240 

Homestead 

250 

Idle cropland stubble 

251 

Idle cropland cover crop 

252 


Idle cropland residue 
Idle cropland' fallow 


“ 253 

254 



ITS's) also have their own assigned grey-scale levels. The 
image on the ground-truth tape is registered to the correspond- 
ing Landsat image. However, the data on the ground-truth tape 
are at a finer resolution. There are 6 subpixels on the ground- 
truth image for each pixel on the Landsat* image. 

In the pixel-level investigations the ground, -truth data are com- 
pared with the AI dot data and with the cluster and classifica- 
tion maps produced by CAMS. This allows a determination of the 
actual composition (in terms of ground-truth classes) of each 
dot, of each cluster on the cluster map, and of each class on 
the class map. Computer programs which do this automatically 
are being developed. 

The schedule for the processing of the pixel-level data is shown 
in figure 6-3. It will be seen that the data for Nebraska and 
Texas will not be processed in time to be included in the final 
report. 
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Figure 6-3.— Pixel-level data processing schedule 























6 . 2 . 5 . 2 . 1 Dot Labeling 

The data set is all the analyst labels for the blind sites, - 
starting April' 1,- 1977, with the ' small fields bias correction 
sheets and .the P.rocedure 1 starting and bias correction' labels- 
recorded in the CAMS packets since June 1, 1977. These 'data 
will have to be key punched by AA for their analyses. 

t 

In order to investigate dot labeling, the composition of each 
dot is obtained ‘first. This consists of determining the repre- 
sentation of the various ground-truth classes (table 6-2)' i ambng- 
the 6 subpixels on the ground-truth tape corresponding to each 
dot. * The composition of each dot will be printed out since it 
is of interest to study the extent to which the- dots 'represent 
mixtures of different subclasses.- Each dot is then labeled with, 
the label of the subclass having the largest representation 
among the 6 subpixels corresponding to that dot oh the ground- 
! truth tape. In what . follows , the term "name" wil-l be used to - 
indicate the ground-truth- label given to a dot in this manner. 
This is to distinguish it from the "label" given to it by the 
analyst-interpreter. It is possible for more than one name- to 
be assigned if two or more subclasses 'have the same -representa- 
tion. The- names are- denoted C m where C denotes' the- nt H subclass 

n n 

and the superscript m denotes the number- of 'subpixel's of this 

class in the dot. The quantity m/6 is called the "purity" of 

the dot since it is. a measure of the extent to wh-ich the dot is 

composed of a single subclass. The term "set" will be used to' • 

indicate the collection of dots named C m . 

n 

Each dot is also given a class name (as distinguished from its 
subclass name) . The classes are those used to label the dots- 
by- the analyst. For Procedure 1, they are expected to be the 
following: spring grains (SG) spring wheat (SW) , winter grains 

(WG) , winter wheat (WW) , grains (G) , wheat (W) , other* (0) , and 
a class denoted X which consists of dots that fell on cloud or 
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cloud shadow and, therefore, were "unidentifiable." These 
classes will be denoted IK,, i- = 1, 8 (i.e.,, = SG, U 2 =• SW, 

etc.). The class'- names are. obtained in the "same way as the 
■ subclass names and are denoted U 3 ^, where :m is the number of 

subpixels of class IK in the dot. . * . 

, * « 

Dot labeling accuracy is studied by estimating two confusion 
matrices — one for classes and one for subclasses. The class 
confusion matrix element P (IK | U 3 ^) is the probability that a 
dot with the name U? will be labeled IK by the analyst. The 
elements for which i •= j give the probability of correct 
labeling for each set- tl^.- For m = 6, one obtains the probabili- 
ties of correct labeling . f.or pure dots;’ -for m < 6, one obtains 
the corresponding probabilities for mixed- dots. The- other 
matrix elements give the probabilities, for confusion with the 
various other classes.-. The .cla-ss confusion .matrix shows how 
well the. .analyst— interpreter is performing at labeling pure, and 
mixed pixels of the- major classes. 


The subclass confusion -matrix element PiU.jc™) is, the probability 

m 3 n 

that a dot with, name C n will be; labeled •Uj. by an analyst. A 
particular subclass C , corresponds to. a particular class U., 
and the probability . of correct labeling-, for the dots named C n ,' 
is given by P (IK , I C™/ ) . The other . elements give the probabili- 
ties for confusion with the various other classes. The subclass 
confusion matrix shows how well the analyst- interpreter is per- 
forming on the various subclasses and-.will indicate any classes 
that pose a particular problem. 

Labeling accuracy- will be a function of- many factors;, a partial 
list of these is given in table 6-3; -The effect of these factors 
on labeling accuracy, will, be evaluated by AA:to - the extent per- 
mitted by available resources . • 
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TABLE 6-3.,- FACTORS AFFECTING LABELING AND 
CLASSIFICATION ACCURACY *' 


Factor 

Description 

■ 

1 

Crop calendar error 

2 

Acquisition history (including biostage) 
Used by analyst 
Used in classification 

3 

Percentage of wheat in segment 

4 

Percentage of small grains in segment 

5 

Percentage of other crops and idle cropland in 
segment 

6 

Percentage of pasture in segment 

7 

Percentage of grasses in segment 

8 

Percentage of irrigation in segment 

9 

Soil type 

10 

Elevation 

11 

Crop moisture index 

12 

Palmer index 

13 

CAMS evaluation code 

14 

Analyst 

15 

Items 3 to 8, historical— 

16 

Field size 

17 

Amount of grazing 

18 

CAMS PCC a for type 1 dots 

19 

CAMS PCC for type 2 dots 


a The CAMS PCC refers to the agreement between the analyst 
dot labels and the assignment of class by the classifier. 
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Finally, a study will be made to determine whether the probability 
of a dot being correctly labeled' is higher if the analyst label 
agrees with the classifier label for that dot. 

6. 2. 5. 2. 2 Clustering 

Three aspects of clustering will be studied: Cluster composition 

and purity, cluster labeling accuracy and the cluster confusion 
matrix. The data set to be used is ail blind site cluster maps 
from April 1, 1977, to the end of Phase -III. • These data cur- 
rently are in the' -form of DTRM tapes- which AA is receiving from 
ISSRS. 

Cluster composition is the' percent of the subpixels in a given 
cluster that belong to each of the major classes. This is 
determined by comparing the cluster map with the image on the 
ground-truth tape. The major classes are SG,’W G, G, 0, and Y, 
where Y consists of both designated other (DO) and designated 
unidentifiable (DU) areas. The set of all these classes will be 
denoted Vj . The "purity" of a cluster is the percentage of the 
total number of subpixels in the cluster that belong to the class 
with the largest representation. The composition and purity of 
clusters are of interest since they ‘indicate' how well the clus- 
tering algorithm is able to separate the classes into relatively 
"pure" clusters. These quantities w'ill be studied as a function 
of segment, region, and acquisition history. 

Cluster labeling will be studied by first naming each cluster 
with the name of the class Vj having the largest representation 
of subpixels in the cluster. , The cluster is correctly' labeled 
if the' label- given by the labeling logic corresponds to this 
name. In the case of nearest-neighbor labeling logic, an 
incorrect label may result from analyst-interpreter mislabeling 
of the dot used to label the cluster or poor performance by 
the labeling logic. If the identity of the dots, which were 


6-44 



used to label each cluster, can be determined, these two sources 
'of error will be studied separately. Cluster labeling accuracy 
will be- studied as a function of cluster purity, acquisition 
history, segment, and region. 


Two ' confusion matrices are estimated for clusters — a class con- 
fusion matrix and a subclass confusion .matrix. The clustering 
confusion matrices will be studied as a function of segment, 
region, and acquisition history. 



that a subpixel of class will appear 1 * in a cluster labeled * " 
Vj . For i = j, this gives the probability of correct clustering; 
for i ^ j , it gives the probabilities of the clustering algorithm 
assigning a subpixel of name Vj to the various confusion .classes. 


The subclass confusion matrix is defined in an analagous fashion. 
The matrix element P {Vj j C n ) is the probability that a subpixel 
of subclass C n is assigned to a cluster labeled Vj*. If is a 
subclass of Vj , the assignment is correct; otherwise, it is not. 
The study of the P(Vj[C n ) allows one to determine if" any particu- 
lar subclass or group of .subclasses- is not getting properly 
clustered. 


6. 2. 5. 2. 3 Classification Accuracy 

Classification will be studied by estimating the classification 
confusion matrices for classes and subclasses. The data set to 
be used is the classification files as the DTBM tapes described 
in the previous section. For classification, the classes are 
SG, WG, G, 0, X, and T, where T indicates pixels which have been 
thresholded by the classifier.' They are denoted Wj , j = 1, 6. 
The subclasses are the same as for dot labeling and clustering. 
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The class confusion matrix element P(Wj[w^) is the probability 
that a subpixel of class is classified as class VT . For 
i = j, this gives the probability of correct classification; 
for i it gives the probabilities of misclassif ication into 

the various other classes'. 

The subclass confusion matrix element’ -P (W^ | C n )- is the probability 

that a subpixel of class C is classified as W., If C is a sub- 

n ^n 

class of Wj , the classification is correct; otherwise, it is not. 
The study of P(Wy| C n >- allows' one to determine if any -particular 
subclass or group of subclasses is not getting properly classified. 

To the extent permitted by avail-able resources, the effect of 
the various factors in table 6-3 on .the classification confusion 
matrix elements will be evaluated. ' - 

6. 2. 5 - . 2. 4 Effect ’of Crop Height and Ground Cover on 
Classification Accuracy . . 

This study will use the crop- height 'and ground-cover data 
acquired -every 18 days for 15 selected wheat fields in each 
blind site. The computer program SPECTL (appendix H) will com- 
pute the probability of correct classification for each of these 
fields and this will be plotted as :a function of crop height, 
Robertson bios.tage, ground cover, and "green number." Means and 
other relevant statistics will be calculated at the segment, 
state, and regional .levels. . . 

6. 2. 5. 3 Pixel-Level Investigations for ITS 1 s 

CAMS will process the 24 ITS's ,in the United States in the same 
manner as. regular LACiE segments are processed. The objective in 
this analysis is to determine labeling and classification accu- 
racies. For winter wheat, this will be done for all three 
procedures used to determine winter wheat proportions, namely 
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the Phase II procedure, the small fields procedure, and Proce- 
dure 1. For spring wheat, only Procedure 1 will be investigated 
because it is the only procedure to be used with spring wheat. 

The ground-truth data consist of two complete inventories of the 
test site, one in the fall of 1976 and the other in the spring 
of 1977. Between these periods, updates are provided every 
18 days, coincident with the satellite overpass, for a subset of 
the fields in the test site. This subset is called the 18-day 
fields. The performance evaluation analysts will use these 
updates to correct the ground-truth designations for the 18-day 
fields on the ground-truth map and keep them current. Also, on 
the basis of the information available for the 18-day fields, 
the analysts will interpret on a current PFC image to update the 
designations for all other fields in the test site; this will 
provide a complete, wall-to-wall ground-truth image for the test 
site. A similar photointerpretation will be performed to obtain 
ground-truth designations for the fields outside the test site 
but within the LACIE segment containing the test site. 

In the Phase II and the small fields procedures, labeling and 
classification accuracy will be determined for each classifica- 
tion for the test site, and a set “of “30 fields chosen at random 
from all the fields in the test site. 

For Procedure 1, labeling accuracy and proportion error will be 
determined for each classification, both for the area within the 
test site and for the whole LACIE segment. 

The "true" proportion of the area within the test site will be 
determined by adding the acreages of the fields 'in the site. For 
the whole segment, the "true" proportion will be determined by 
applying a bias correction, based on ground-truth labels, to the 
machine proportions of the segment. 
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The results of' these investigations will' be reported in the 
interim reports -and at least one ITS- story Will be presented in 
the quick-look following the CAS CMR. Thus, the ITS can fulfill 
^a role not possible by the blind sites —‘illustration of accu- 
racy using annotated imagery. 

I • * 

iThis effort ’’starts February 1977 when the first inventories are 

i 

received 'and ends in October -when the harvest inventory is 
received. - ' " 

6. 2. 5. 4 Second-Order Investigations on the IMAGE 100 System 

1 »* , , , 

These investigations are designed to evaluate results obtained 
with the IMAGE 100 Hybrid Procedure 1 . operational system. 

6. 2. 5. 4.1 Study of Proportion Error 

The data set will consist, of 24 ITS's in the United States and 
10 ITS's in Canada. The IMAGE 100 proportions will be determined 
for the test sites (which are smaller than the whole segment) and 
compared with the corresponding .ground-truth proportions deter- 
mined in the manner described, in .section, 6. 2. 5. 3. The method 
used to make the comparison .is .described in section 6. 2.4.1. 

6. 2. 5. 4. 2 Effects of AI, Acquisition History, and Bias 
Correction on Proportion Estimation Error 

The IMAGE 100.. processor and data from eight U.S. blind sites will 
be used in an experiment wherein -each site will be analyzed by 
three AI's to give a Procedure 1 "raw" and a "bias-corrected" 
estimate of the proportion of small grains in each segment. The 
segments .will be of two -types; namely > those having acquisitions 
in all four biophases and those- having .only early season acqui- 
sitions. The segments will be cho.sen at random from the blind 
sites for which detailed ground-truth . data are available. 
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The objectives of -the experiment are: (1) to evaluate the- perform- 

ance of Procedure 1 in terms of absolute proportion . estimation 
error and its repeatibility over AI r s , (2) to make comparisons 

between "bias-corr.ected" and "raw" Procedure 1 estimates , .-and 
(3) to determine if the performance is better when acquisitions 
from all biostages are used than when only the early season 
acquisition is used. 

Analysis of variance will be used to test for the effects of 
Al's-, -time (i.e., early season versus all acquisitions), method 
(raw versus bias correction estimates), and their interactions. 

6.2.6 ACCURACY ASSESSMENT EVALUATION OF THE NEW SAMPLE STRATEGY 

During LACIE Phase III a new sample strategy is being tested 
over Kansas and North Dakota. AA evaluations are designed to 
examine how well the new sampling strategy is doing over these 
areas through comparisons between the old and new strategies . 

The evaluations utilize (a) blind site LACIE .classification and 
ground-truth data collected during the 1976-1977 growing season 
over Kansas, and North Dakota, (b) 1976-1977 LACIE and USDA/SRS 
production, area, and 1 yield estimates over these .same states, 
and.(c) 1974 USDA/SRS historical county data also for Kansas and 
North Dakota; 

The"AA evaluations of the new sample strategy consist of: 

a. Subtask 1 making comparisons (relative differences and CV's) 

, between the 1977 LACIE estimates of production, area, and 

yield over Kansas and North Dakota from the old' and new 
sample strategies, and estimates made by USDA/SRS. Fig- 
ure 6-4 further defines ^these comparisons. 

b. Subtask 2 determining and comparing aggregated acreage bias 
for old and new sample strategies over Kansas (winter wheat) 
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SUBTASK 1 


LACIE AREA, YIELD 
(FROM (CEA1 YIELD 
MODEL), AND 
RELATED PRODUCTION 
1977 ESTIMATES 
FROM 

• 81 KANSAS 
(NSS SEGMENTS) 

• 83 N. DAKOTA 
(NSS SEGMENTS) . 


COMPARE 

• RELATIVE 
DIFFERENCES 

• COEFFICIENT OF 
VARIATION 


COMPARE PRODUCTION, 
AREA, AND YIELD . 
USING' 

• RELATIVE DIFFERENCES, 

• COEFFICIENT OF 

VARIATION- , "■ ' 


SRS 1977 
ESTIMATES 
.OVER 

• KANSAS 

• N. DAKOTA 


I 


LACIE PRODUCTION, 
AREA, AND YIELD 
1977 ESTIMATES 
FROM 

• 121. KANSAS 
(OLD SS) 

• 103 N, DAKOTA 
(OLD SS) 


-\ 


LACIE AREA, YIELD 
(FROM FEYERHERM 
MODEL), AND RELATED 

4 . fc 


COMPARE PRODUCTION, " 

SRS 1977 

PRODUCTION 1977 

AREA, AND YIELD 

ESTIMATES 

ESTIMATES FROM 

USING 

OVER 

• 81'KANSAS 

• RELATIVE DIFFERENCES 

• KANSAS 

(NSS SEGMENTS) 

• COEFFICIENTS OF VARIATION 

• N. DAKOTA 

• 83 N. DAKOTA 
(NSS SEGMENTS) 




r 


THIS COMPARISON IS PART 
OF THE REGULAR AA 
.EVALUATIONS • ' 




TO DETERMINE 
DIFFERENCES 
BETWEEN 
> THE NEW 
AND OLD 
SAMPLE 
STRATEGIES 


“I 


SRS 1977 I 

ESTIMATES , 

OVER I; 

• KANSAS I 

• N. DAKOTA j 

I 


SUBTASK 2 

- 






DETERMINE D* 



KANSAS D 



^AVERAGE DIFFERENCE, 
P _ p Q1 !) FOR 
N. DAKOTA FROM 20 
N. DAKOTA BLIND SITES 

* . 

1 

r 


J 

- 

• 

AGGREGATE WITH 
KANSAS D INSERTED 
FOR EACH OF 81 
KANSAS SEGMENTS 
TO GET AGGREGATED 
ACREAGE BIAS ESTIMATE 
FOR NEW SAMPLE 

COMPARE TO 
EXAMINE 

AGGREGATE WITH 
KANSAS D INSERTED 

*D WILL BE INITIALLY 
CALCULATED FROM 
PHASE II BLIND SITES 
AND PLANTED PHASE III 


NSS - OSS 
BIAS DIF- 
FERENCES 
OVER KANSAS 

-4 W 

FOR EACH OF 121 KANSAS 
SEGMENTS TO GET 
AGGREGATED ACREAGE 
BIAS ESTIMATE FOR 
OLD SAMPLE STRATEGY 

BLIND SITE DATA 


STRATEGY 


^ w 


DETERMINE D* 
(AVERAGE DIFFERENCE, 
f - P GT 1 for 

KANSAS FROM 30 
KANSAS BLIND SITES 


AGGREGATE WITH 

COMPARE TO 
EXAMINE 

AGGREGATE WITH 

N. DAKOTA D INSERTED 

NSS - OSS 

N. DAKOTA D INSERTED 

FOR EACH OF 83 N. DAKOTA 

BIAS DIF- 

FOR EACH OF 103 

SEGMENTS TO GET AGGRE- 

FERENCES 

N. DAKOTA SEGMENTS 

GATED ACREAGE BIAS 

' OVER 

TO GET AGGREGATED 

ESTIMATE FOR NEW SAMPLE 

N. DAKOTA 

ACREAGE BIAS ESTIMATE 

STRATEGY ■ 

« ► 

FOR OLD SAMPLE STRATEGY 


Figure 6-4.— AA new sample strategy (NSS) evaluations. 
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and North Dakota (spring wheat) . • Figure 6-4 also provides 
a:n additional description of these evaluations. 

«*■* , ^ ‘ i' " ' ' > » . . 

c. Subtask 3 performing regression analysis using 1977 blind 
site classification data, ground-truth data, and USDA/SRS 
1974 historical census data to obtain three basic regressions. 

® True segment proportion vs . historical stratum proportion 

' • « LACIE segment proportion vs. ground truth segment “ 

■ proportion 

0 LACIE segment proportion vs. historical stratum proportion 

Using the results of these regressions, a sensitivity analysis is 
performed in which the acreage error is quantified (a) with both 
the sampling and the classification errors, (b) with the sampling 
error and no classification error, and (c) with the classification 
error and no sampling error.. Then the reduction in error from 
(a) - (b) and (a) -(c) is examined to establish which, if either, 
is contributing more error to the acreage estimate. 

6.3 ' AA REPORTING 

There are three types of AA reports: Special management brief- 

ings, monthly quick-look reports, and interim and final reports. 

6.3.1 AA SPECIAL MANAGEMENT BRIEFINGS 

i ’ 

As required to support LACIE reporting functions, special brief- 
ings are prepared for LACIE project management relative to the 
current status of LACIE operational data development, particu- 
larly with respect to special problems that could affect the 
accuracy of final LACIE at-harvest estimates of wheat acreage, 
yield, and production in the USGP. 

In addition, AA personnel support LACIE project management 
requirements to' brief RASA, USDA, and NOAA upper-level manage- 
ment on the status of LACIE outputs and the progress being made 
toward satisfying the 90-90 criterion. 
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6.3.2 AA MONTHLY QUICK-LOOK REPORTS 

The quick-look reports contain an evaluation by AA of the LACIE 
estimates reported in the CMR's and the CAR. The quick- look 
reports are released one week following the release of a CMR or 
a CAR. The CMR's and CAR's contain the official LACIE estimates 
of wheat production, area, and yield, and the corresponding 
statistics. The true wheat production, area, and yield for the 
particular region or country are, of course, unknown. Therefore, 
to ascertain the accuracy of the LACIE estimates, comparisons 
are made with a reference standard. In the United States, the 
reference standard consists of the most recent (at the time of 
the comparison) estimates released by the USDA/SRS. In foreign 
i countries, the reference consists of the most recent estimates 
released by the USDA/FAS. 

In addition, the quick-look reports will contain (a) significance 
tests of no difference (between the LACIE estimates and the ref- 
erence standard) at the region or country level; (b) results of 
blind site analyses of proportion estimation error; (c) classi- 
fication bias aggregated to the regional level; and (d-) within 
stratum acreage variance due to classification and sampling 
errors. 


6.3.3 AA INTERIM AND FINAL REPORTS 

The interim reports are released at regular intervals throughout 
the crop season. They contain the results of the previous quick- 
look reports, and the results of all other AA investigations up 
to the time each report is written. 

-? * « - 

Each interim report is built up from the previous one by includ- 
ing data that became available during the interim period. Tech- 
nical comments on each report are solicited from a variety of 
sources and are used to upgrade subsequent reports. Early and 
mid-season evaluations are made in the first and second interim 
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reports; late season and at-harvest evaluations are made in the 
third and fourth interim reports. 

. The fourth interim report also serves as a draft for the final 
! report, which contains material similar to- the interim reports 
but covers the entire year. 

The planned schedule for release of the LAC IE interim AA reports 
and the final Phase III AA report is as follows: 

a. First interim report - May 15, 1977 

b. Second interim report - August 15, 1977 

c. Third interim report.- November 15, 1977 

d. Final interim report - March 1, 1978 

e. Final Phase III AA report - June 1, 1978 

The planned contents for the four interim reports and the final 
report are shown in figure 6-5. The AA interim reports require 
approval by the AA manager and the Chief of the RTEB of NASA/ JSC 
prior to their release for LAC IE project review and evaluation. 

The final LAC IE Phase III AA report requires review and approval 
of the following persons prior to its release for distribution: 

a. D. E. Pitts, Manager, LACIE AA, NASA/ JSC 

• b. J. D. Erickson, Chief, RTEB, NASA/ JSC 

c. J. L. Dragg, Chief, Applications System Verification Branch, 
NASA/JSC 

d. F. G. Hall, LACIE Project Scientist, NASA/ JSC 

e. J. Hill, Assistant Deputy Project Manager for LACIE, NOAA 

f. J. Murphy, Assistant Deputy Project Manager for LACIE, USDA 
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FIRST INTERIM REPORT (MAY 15.197?) 


SECOND INTERIM REPORT (AUGUST 15,1977) 

- 

THIRD INTERIM REPORT (NOVEMBER 15,1977) 





DATASETS 




DATASETS 


OATASETS 

• US. AND USSR CMR'S (THROUGH END 
OF YEAR) 

• CAR 

• GROUND TRUTH (HARVESTED) 

- 135 WW BLIND SUES 

- 65 SW BLIND SITES 

• 1977 SRS STATE ESTIMATES 

• 1977 USSR ESTIMATES 

k. 





FEBRUARY AND OCTOBER! 

• GROUND TRUTH (HARVESTED) 

- 94 WW BUND SITES 

- 59 SW BUND SITES 

- HITS 

• 1977 SRS STATE ESTIMATES 

• 1977 U.SS.R ESTIMATES 




MARCH, AND APRIU 

• GROUND TRUTH (PLANTED) 

- 45 FAIL WIN BL1N0 SETS - USSGP 

• 1 977 SRS STATE ESTIMATES 


APRIL, MAY, JUNE, AND JULY) 

• GROUND TRUTH (PLANTED) 

- 45 FALLYW9UND SITES 

• 1977 SRS STATE ESTIMATES 


r 



EVALUATIONS 




EVALUATIONS 


EVALUATIONS 



• IACIEVS SRS ESTIMATES COMPARISONS 

- 1477 ESTIMATES OVER S STATE WW IIEGION 

- 1977 LAClE COMPARISONS WITH 1976 IACIE 


• COMPARISONS 

- 1377 IACIE VS 1977 SRS ESTIMATES 
OVER 7 -STATE WW REGION 

- 1977 LAClE COMPARISON WITH 1976 IACIE 




REPORT CONTENTS ARE SAME AS FOR THE THRID 
INTERIM REPORT 


REPORT CONTENTS ARE THE SAME AS FOR 
THE FOURTH INTERIM REPORT 

• SO 90 CRITERION EVALUATIONS OVER 5 STATE 
WW AREA 


• 90 90 CRITERION EVALUATIONS OVER 5 STATE 
WW AREA 


• 90 90 CRITERION EVALUATION OVER 9 STATES 


REPORT CONTENTS ARE THE SAME AS FOR 
THE THIRD INTERIM REPORT 



• FIRST ORDER ERROR SOURCE EVALUATION 
FOR S STATE USSGP 

- WHEAT (Pfl) PROPORTION ERROR 

- CLASSIFICATION VARIANCE 

- SAMPLING VARIANCE 

* 

• FIRST ORDER ERROR SOURCE EVALUATIONS 
FDR 5 STATE USSGP 

- WHEAT |Pfl) PROPORTION ERROR 

- CLASSIFICATION VARIANCE 
< - SAMPLING VARIANCE 


• FIRST ORDER ERRORSOURCE EVALUATIONS 
FORUSGP 

- WHEAT PR OPO RTI 0 N E R R OR 

, - SMALL GRAIN PROPORTION ERROR 

- CLASSIFICATION BIAS 

- CLASS! FI CATION VARIANCE 

- SAMPLING VARIANCE 
-PROCEDURE 1 HYBRID ACCURACY 


• FIRST ORDER ERRORSOURCE EVALUATIONS 
FORUSGP 

REPORT CONTENTS ARE THE SAME AS FO R 
THE THIRD INTERIM REPORT 



• SECOND ORDER SEGMENT LEVEL EVALUATIONS 
- SPECIAL STUOIES 


■ • SECOND ORDER SEGMENT LEVEL 
EVALUATIONS 

- THRESHOLDING EFFECTS 

- SPECIAL STUDIES 

- 

* SECOND ORDER SEGMENT LEVEL EVALUATIONS 

- CAMS EVALUATION CODE 

- CROP CALENDAR ERROR 

- PROCEDURE 1 HYBRID ACCURACY 


• SECOND ORDER SEGMENT LEVEL EVALUATIONS 

- CAMS EVALUATION CODE 

- HAZE EFFECTS 

- PROPORTION ERROB VS BIOSTAGE 

- CROP CALENDAR ERROR 

- PROCEDURE t HYBRID ACCURACY 


.«> 

, v ' 

• SECOND ORDER PIXEL LEVEL EVALUATIONS 

NO PIXEL LEVEL EVALUATIONS IN THIS 
REPORT 

- 

• SECOND ORDER PIXEL LEVEL EVALUATIONS 
OF LABELING, CLASSIFICATION, AND 
CLUSTERING ACCURACY 

NO PIXEL LEVEL EVALUATIONS IN THIS 
REPORT 


• SECOND ORDER PIXEL LEVEL EVALUATIONS 
OF LABELING, CLASSIFICATION AND 
CLUSTERING ACCURACY . 

- 20 BUND SITES 

- WHEAT VS OTHER 

- SMALL CRAIN VS OTHER 

- 80UNOARY PIXEL LABELING ACCURACY 

- 15 FIELD ANALYSIS 

- PROCEDURE 1 HYBRID ACCURACY 


« SECOND ORDER PIXEL LEVEL EVALUATIONS 
OF LABELING CLASSIFICATION, AND 
CLUSTERING ACCURACY 

REPORT CONTENTS ARE THE SAME AS FOR 
THE THIRD INTERIM REPORT, * 


v 



Figure 6-5.— Summary of Phase III AA interim and final reports. 






























g. R. B. Erb, Chief, LAC IE Project Office, NASA/ JSC 

h. R. B. MacDonald, Chief, Earth Observations Division, NASA/JSC 
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APPENDIX B 


LAC IE PHASE III BLIND SITES, TEST SITES, 

AND INTENSIVE TEST SITES 

The LAC IE ITS and blind-site selections' are given in the follow- 
ing tables. 


Table 

Sites 

B-l 

LAC IE Phase III U.S. ITS 1 s 

B-2 

LAC IE Phase III Canadian ITS 1 s' 

B-3 

Phase III Test Site and Blind-Site 


Distribution • 
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TABLE B-l.— LACXE PHASE III U.S. INTENSIVE TEST SITES 


Segment 

State 

County 


Site 

size 

Wheat 

type* 

Acquired 

as 

Lat. , N 

Long. , W 

N. mi. 

Km 

1961 

Kansas 

Morton 

o 

o 

kb 

H 

0 

l> 

101° 54 ' 00 " 

"5x6 

■ 

WW 

ww 

1962 

Kansas 

Saline 

38°41’48" 

97° 28 ' 24 " 

3x3 


ww 

ww 

1963 

Kansas 

Rice • 

38°17'00" 

98°12 ' 42" 

3x3 

5. 5x5. 5 

WW 

ww 

1964 

Kansas 

Ellis 

38°50'06" 

99°13 1 00" 

3x3 

5. 5x5. 5 

ww 

ww 

1988 

Kansas 

Finney 

38° 10 1 12" 

100°43 1 12" 

5x6 

9xll 

ww 

ww 

1965 

N. Dak..' 

Burke 

48°53 ’ 12" 

102° 10 1 00" 

5x6 

9x11 

SW 

SW 

1965 

N. Dak. 

Williams 

4 8 0 1 9 1 1 2 “ 

103°24 '42" 

5x6 

9x11 

SW 

SW 

1967 

N. Dak. 

Divide 

4 8 °53 ' 36" 

103°iq’54" 

2x10 - 

3.7x18.5 

SW 

SW 

1969 

Montana 

Toole 

48°53'00" 

111°46 ' 36" 

2x10 

3.7x18.5 

s&ww 

SW 

1970 

Montana 

Liberty 

48°44'00" 

110°51'00" 

2x10 

3.7x18.5 

s&ww 

SW 

1971 

Montana 

Hill 

4 8 ° 4 2 r 00" 

109°55 *00" 

2x6 

3.7x11 , 

s&ww 

SW 

1973 

Wash. 

Whitman 2 

46° 50 1 24 " 

117°48 ' 18 " 

3x3 

5.5x5. 5 

s&ww 

ww 

1975 

Idaho 

Oneida 

42°04 1 30“ 

112°29 1 30>' 

3 x 3 " 

5. 5x5. 5 

s&ww 

ww 

1976 

Idaho 

Franklin" 

42°08 ' 00" 

lli°58'00" 

3x3 

■ 5. 5x5. 5 

s&ww 

WW 

1977 

Idaho 

Bannock 

42°56 ’ 30" 

112°25 ’50" 

3x3 

5. 5x5. 5 

s&ww 

ww 

1978 

Texas 

Randall 

35°09'30" 

102°04 ' 24 " 

3X3 

5. 5x5. 5 

ww 

ww 

1979 

Texas 

Deaf Smith 

34° 52 ' 12" 

102°22 ’18" 

3x3 

5. 5x5. 5 

ww 

ww 

1980 

Texas 

Oldham 

35°15' 00" 

102° 32 ’00" 

3x3 

5. 5x5. 5 

ww 

ww 

1981 

Indiana 

shelby 

39°27 1 36" 

85°47 '12" 

3x3 

5. 5x5. 5 

ww 

ww 

1982 

Indiana 

Madison 

40 °13 ' 30" 

85° 37 ' 50“ 

3x3 

5. 5x5. 5 

ww 

ww 

1983 

Indiana 

Boone 

4 0 0 05 ' 42" 

86° 33 ' 90 " 

3x3 

5. 5x5. 5 

ww 

ww 

1687 

S. Dak. 

Hand 1 

4 4 ° 3 5 '0 0 " 

98°58 ' 00" 

5x6 

9x11 

s&ww 

SW 

1986 

S. Dak. 

Hand' 2 

44 ° 21 ' 00" 

98°45 r 06 M 

5x6 

9X11 

s&ww 

SW 

1987 

Minn. 

Polk 

4 7 0 4 9 1 00" 

96°41 1 00" 

5x6 

9xll 

SW 

SW 


* As indicated by ground- truth data: 


WW = winter wheat 
SW = spring wheat 
S&WW ~ spring and winter wheat 


o* ?°° 
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TABLE B-2 LACIE .PHASE III CANADIAN INTENSIVE TEST SITES 


Segment 

Province 

County 

Center coordinates/site size 

Wheat 

type* 

Lat. , N 

Long., W 

S. mi . 

Km 

1958 

Saskatchewan 

'Mel fort 

52°48 ' 

104044 ' 

2x10 

3.2x16 

SW 

1959 

British Columbia 

Dawson Creek 

55°48 ' 

120?12 ' 

2x10 

3.2x16 

SW 

1984 

Saskatchewan 

Delisle 

51°55 1 

107"28' 

2x10 

3.2x16 

SW 

1985 

Saskatchewan 

Swift Current 

50°19 1 

10?o53' 

2x10 

3.2x16 

SW 

1989 

Alberta 

Lethbridge (Raymond) 

49030 ' 

112“48 ' 

2x10 

3.2x16 

SW 

1990 

Manitoba 

Stony Mountain 

50° 04 1 

97 "21 1 

2x10 

3.2x16 

SW 

1991 

Manitoba 

Starbuck 

490471 

97"29 ' 

2x10 

3.2x16 

SW 

1992 

Alberta 

Olds^ 

51°54 ' 

113 "32 ' 

2x10 

3.2x16 

SW 

1994 

Alberta 

Ft. Saskatchewan 

53° 38 1 

113 " 07 ' 

2x10 

3.2x16 

SW 

1995 

Manitoba 

Altona 

49012 ' 

97° 38' . 

1*5 

1.6x8 

SW 


*As indicated by ground-truth data: SW = spring wheat. 
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TjKB'L'E B-3 . — PHASE III TEST -SITE AND 
BLIND-SITE DISTRIBUTION . 


. State ■ 

- -Number- -of- sites 

t 2 , / 

Colorado ' ' 

15' 

Kansas 

31 

Nebraska 

26 ‘ 

Oklahoma 

20 

Texas 

is 

Minnesota 

20 

Montana 

29 

N . Dakota 

30 

S . Dakota 

23 

Total 

212 

Canadian Test 
Sites 

30 
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APPENDIX C 


CONTRIBUTION OF SAMPLING AND CLASSIFICATION TO ACREAGE 

. ESTIMATION ERROR 

This appendix describes the calculation of the contribution of 
sampling 1 and classification errors to the variance of the LACIE 
production estimate. ‘ 


C. 1 APPROACH 


The variance o.f._the- LACIE acreage estimate- for a large area 
(e.g., zone) can be written 


2 2 

:v - Z v.a. 

i i i 


( 1 ) 


where 0^ is the variance of the acreage estimate for the i th 
county and is a weight which depends on the size of the 
county, the number of segments in the county, etc. (Refer to 
the CAS Requirements Document, appendix B for details.) 

2 

The variance 0^ represents a mean-squared deviation between the 
LACIE estimate for the county wheat proportion and the true 
county wheat proportion. This variance is caused mainly by 
two factors: sampling errors and classification errors. 


In accuracy assessment, it is desirable to -quantify the contribu- 
tion of each of these error sources to the large area production 
estimate. The LACIE production estimate depends on acreage and 
yield estimation errors in a complicated way; hence, it is 
unrealistic to assume the error in the production estimate can 
be written as a sum of uncorrelated random variables representing 
acreage and yield errors. Instead, the effect of a particular 
error source is measured by the reduction in the LACIE production 
variance which would be achieved if that source were eliminated. 


C-l 



It will be assumed that the 1th county acreage error variance 

2 . 2 2 2 2 - 2 
a. can be written a. - a + X a , where o is a contribution due 
X X C S c 

to classxfication, and X a is . a contribution due to sampling. 

To determine the effect of no classification error, the variance 

of the LAC IE production estimate will be made using pa? instead 


of a^ where 


p is an estimate of the ratio 


,2 2 
X 0 s 


2 22* Similarly , 

0 + X a 

c s 2 

the effect of no sampling error is estimated by replacing a. by 
2 ^ 

(1 - pja^. The following. »two- sections describe the methods 

employed for estimating sampling and classification variances 

and the function p. 


C. 2 ACREAGE REGRESSION MODELS 

For counties with one sample segment, the LAC IE estimate of the 
1th county wheat proportion can be written 


y. = C. + 
-’'x - x 


(x. - C . ) . + (y . - x. ) 
x~ l ■ • 1 


«* C. + e. + 6. . (2) 

x . x , x 

where ■ -■ *"'■ - 

- w u. * '■ 

y^ = LACIE estimate of the wheat proportion in the sampled segment 

C^ - true {current year) proportion of wheat in the county 

x^ = true proportion i of wheat in the sampled segment 

= sampling error = x^ - C^ 

6. = classification error = y. - x. 
x J x x 
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It will be assumed that for some reasonably large area (e.g., a 
zone) the errors and 6^ have the following properties: 

e. and <5. .are uncorrelated , (3) 

_1_ 

i 

E{e ± ) = 0 (4) 

E(6 i |x i ) = X*x. + 0 (5) 

V(e A ) - g 2 s (6) 

V (5 . | x • ) = a 2 . (7) 

i x c 

It is also assumed that there is a linear model relating the 
current year counties proportions, c^, to the historical propor- 
tions, which' will be denoted by Z^.; i.e., > ‘ 

C . = a + 8Z • + ? • (8) 

1 x ^x 

2 

where E(£^) = 0, V(£^) = Cov(C^,£^) = Cov(?^,6^) = 0 and 

a and '8 are regression coefficients. 


Prom" the above, assumptions' and definitions, three basic 
regression models are obtained; 

a. True segment proportion vs. historical county proportion — 

from the definition of e . , 

x 


x i ” c i + e i 


It follows that 


a + 8Z . + £. + e . 
x ^x x 


(9) 


B(x. ) - a + 8Z . 

**» » JL 

v( *i> - 4 + °s 


( 10 ) 


b. LAC IE segment proportion vs. ground truth segment proportion - 

f 5. 
x 

y . = x . + 6 . . 

J x x x 


from the definition of 5. 

x 
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a to * a to w to 


It follows that 


E (y . lx. ) = x. + X*x. + 9 

3-- ' >L ' X ^ ^ X 

V(y i lx i ) = ol 

Writing X = 1 + X*, one obtains 

E ( y il x i> ~ Xx i + ' 6 ' 

V(y i |x i ) = a 2 c ;~ - ; 


(11) 

(12) 


c. LACIE segment proportion vs. historical county proportion — 
from equations (9) through (12) , 

E (y . ) = E rE(y.jx-)*]= E„‘ (Xx.. + 0) s‘=.X(a-.+ £Z.) + 0 (13) 

i i 1 - 1 1 - 1 x 

v< y± ) = E x.C v( yii x i ) ] + v x.[ E( yii x i>] = a l + x2 (°h + °l) 


(14) 


,2 2 
X a s 


As stated previously, one would like to estimate- p = — ^ 2~2 * 

a + X <j„ 
c s 

. “ i 

None of the three regression models permits an estimate of 

2 2 2 2 • 
a g separately from i.e., one can only estimate + a not 

alone. If current year county proportions were available, 
could be estimated, but since this is not the case, 

x 2 ( 0 ; 


2 '- 2 - «£) 


2 . , 2/2 , 2 \ 

o + X (o + 0 H ) 


will be estimated instead of p. If 


<< vf. (a reasonable assumption) then p*. « p. 
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C. 3 HORMALITY ASSUMPTIONS - MAXIMUM LIKELIHOOD ESTIMATION OF p* 

Suppose a given zone, has m- blind site’ segments and n ordinary (i-.e., 
not blind site) segments, and let the blind site segments be num- 
bered 1 to m. It is assumed that. ground, truth wheat proportions 

t 1 

ITi 1 * ' * 

■C x i^'i=l are available tor the blind sites and LA.CIE estimates 

are available for all the segments. It is also assumed 

that historical wheat proportions are available for the 

counties containing the segments. If << a , so that p » p* 

the regression models equations (9 through 14) can be used 
to obtain 

E (x . ) ~ a + BZ ; v (x. ) 4= cl 

■1 J. X D 


E(y i^ x i^ " Ax i ' + 0; V ^ Y il x i^ 



I “ 

1/2,* 

• • ,m 


i = 

1,2,* 

• * ,m 

c< 

CN 

2 


m+1 , 

A a s 

+ a 

c 

i = 


If there is one segment per county, then the errors and 6^ 
are independent for different values -of i, and hence the likeli- 
hood function of the sample can be written . - 


m m+n 

L= n f(x i' y i > n h(y i> 

i=l i=m+l 


(15) 


where f(x. ,y.) is the joint density of x. and y. for i = l,m and 

x 1 x X 

h(y^) is the density of y^ for i = m+l,m+n. 


m m 

The function J~J f (x^,y^) can be written n f ( x i' y i) = 

• i«l ' i-1 

m . . 

jj[ f(yj_lx^) g(x^) where fCy^lx^) is the conditional density 
i=l * - 

of y^ given x^ and g{x/) is the- density of x^. 
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• m 


m 


If normality is assumed/ I! f (x djf y i ) i = JJ — 


i=l 


. , a /2ir 
i=l c 


exp l'^ E (y i ■ Xx i ■ 8 ’ 2 }^5i eXP i‘ ^ E (X i " 0 ■ 62 i )21 


-2<J - 

S 1=1 


and 

m+n 


' m+n 


n +1 h<yi> = e T 2 (xm- + 


rr . E (y i - Xa 


- 0 - XgZ ± ) 


Letting Q = -21ogL - log2ir , 


2 2 , 


D T 
m , m 


T 


Q = m log + m log a + n log (a + X a ) + — k- + 


n 


2 2 2 , -.2 2 
a cr cr + X o 
CSC s 


(16) 


where 


m 

D = E (y. 

m ! i 


Xx i - 61 


m 

T n.-f.< x i - “ - B z i> 


m+n • o 

* n = J2 ( Yi - xa - e - xsz ± ) 

-i=m+l 


One attempts to maximize L by finding a stationary point of Q: 


1 3Q _ 1 1 

2 3a 


m 

E (x. - a - (3Z 


m+n 


j .> X(y ^ - Xa - 6 X&Z^) 

1 t 1 


m+1 

+ 2 7TTT 

a + X a 


= 0 


(17) 
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1 3Q 

2 30 


m 

2 Z . (x. -■ a 
1 ■ 


m+n 

- 0Z. ) ' X3 XZ i (y i ~ Xct '~' 9 
1 m+1 


X6Z.) 


2 


a 2 + xV 

c s 


= 0 


( 18 ) 


1 3Q 

2 30 


m- 

2 (y. - Xx. 
1 • x x 


- 0) 


m+n 

<Y ± - Xa - 0 

m+1 


X0Z i ) 


a * 


2 , ■> 2 2 
a + a a 
c s 


= 0 


(19) 


m , 2 I 2t? 

2 x. (y. - Ax. - 0) “ nXa s + V' + a) (y i 

1 3Q _ 1 1 1 1 . i=ttl 

2 3 A - + = — 


- Aa - 0 


xez ± ) 


2 , TTT 
a c + X a s 


A 2 a 2 T 
s n 

(2 , ,2 2\2 
(°c + X a sj 


= 0 ' 


( 20 ) 


9Q . 
3a 2 


m 


n 


X 2 a 2 + 
s 


m 

4 

c 


n 


(> 2 °l -If 


= 0 


(21) 


3Q_ 

do' 


m 


nA‘ 


i 2 2 

X a s + 


m 

4 

s 


V 


(°l + 


= 0 


( 22 ) 


Equations (17) through (22) must be solved for the parameters 
r 8/ 8/ X r o^ f and Ug. If a, 0, 0, A, a 2 / and a 2 represent the 
solution to .equations (17) through (22), then the invariance 

theorem for maximum likelihood estimation can be used to 
obtain 


P = 


X 2 $ 2 

s 


-2 , P2 A 2 

O + A cj 

c s 


( 23 ) 


as the maximum likelihood estimate of p . 
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The equations (17) through (22) are nonlinear but can be 
solved using /numerical techniques. Newton's Method was used to 
solve the equations for this report? • i.e.,' if u (k) is an estimate 
of the solution vector .u = (a, £, 0, %, ol, oh at the k th step, 
then 


u (k + D . u (k> . F -i f ( u (k)) (24) 

where f(u^ = (f^, • • • ,fg) ^ is the vector of the left sides of 


equations (17) through (22) evaluated at u 
3f . 


(k) 


and F = (F . . ) 

ID 


In practice, it was slightly simpler to use the parameter 
transformations 


r = 


0 

s 

7T~2 7 2 

A 0 +0 

S C 


(25) 


and 


e _ ,22 , 2 

S — A 0 i (J 

s c 


(26) 


and solve for a, 8, 6, X, r, and s. Again, the invariance 
theorem can be used to give 


A AO A 

p = a r 


(27) 


C. 4 ACCURACY OF p 

Since p is an extremely complicated function of the data, it is 
impossible to write- down.-'the- variance of. p for finite sample 
sizes m and n. However, the asymptotic variance of p can be 
estimated using the information matrix; i.e., if 


T 



(28) 



<2.-3- 


and g(u) = g^a,3,6 ,£,a^,a^ j is a differentiable function of the 
parameters u, then the variance of g(u) is asymptotic to 


[g 1 (u) ] T V 1 g* (u) 

q * ( u ) r= /_^£ . . . 

9 1 ’ K' 'a»J 


where 


(29) 


Thus, in our case, g(u) = -g— 2" 

X a + a 
s c 


g 1 (U) = 


0,0,0, - 2Xa i s (x 2 o 2 s + °l)~ 2 ,-ol(x 2 o 2 s + a 2 c ) ^ + \z a ^ 


,2 2 

* °s 


(°l ♦ ^ IV 


(30) 


To estimate x, the observations {x^}, {y^}, and {Z^} and the 

/ A A A A A 2 a2 \ 

estimated parameters (a f 3 / 0,X,cr f and a^J were substituted into 

2 

the matrix H » (h, . ) = 1 ° < ? L . Then equation (29) was used 

i 3 

to obtain an approximate variance for p. 
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APPENDIX D 

INSTRUCTIONS TO USDA/ASCS 
■FOR 

• LACIE- BLIND SITE 
SEGMENT INVENTORY 

• LACIE BLIND SITE 
15-FIELD PERIODIC 
OBSERVATIONS 





REPLY TO 
ATTN OF: 





national aeronautics and space administration . 

LYNDON D. JOHNSON SPACE CENTER 
Houston, Texas 77058 


TF4-76-11-31 


NOV 2 2 1976 


, 0 UJT/0/\, 

& % 





TO: County Executive Director 

FROM: Bobby E. .Spiers 

Production Specialist 

SUBJECT: Transmittal of Material for LACIE Segment Inventory 


Enclosed is the material to be used to complete the identification of 
crops in the LACIE segment. located in your county. The information 
requested is essential to the evaluation of the effectiveness of our 
experiment and v/e ask that you make the inventory and return the data 
to us within- two weeks. 


The instructions explain in detail what- needs to be done. Read 
instructions and if you have questions, let us know. Thank you 
your assistance. 


Bobby E. Spiers 





the 

for 


Enclosures 

1. Instructions 

2. Post Card 

3. Return Label 

4. Color Infrared Print(s) 
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FALL ‘ 

INSTRUCTIONS 

FOR' 

LAC IE SEGMENT INVENTORY- 
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I. Introduction 
A. Background 

The LACIE (Large Ar.ea Crop Inventory Experiment) is an inter- 
agency experiment in the use of Landsat (formerly called Earth Resources 
Technology Satellite) and meteorological data to identify and inventory 
crop production. Participating agencies include the Department of 
Agriculture, the National Aeronautics and Space Administration, and 
the National Oceanic and Atmospheric Administration. Within the 
Department of Agriculture, participating agencies are the Agricultural 
Stabilization and Conservation Service, Economic Research Service, 

Foreign Agricultural Service, Agricultural Research .Service, Soil Con- 
servation Service, and Statistical Reporting Service. The overall 
general objectives of the LACIE are to determine utility and cost 
effectiveness of satellite and surface derived data sources to monitor 
large area crop (wheat) production and assess the impact of agricultural 
and meteorological conditions on production estimates. The utility of 
the information produced will be evaluated on the basis of its objec- 
tivity, timeliness, accuracy, and its expected value for policy and 
program decision making. 

LACIE reports are based on data extracted from 5 x 6 mile segments 
that have been randomly placed throughout the wheat producing region of 
the United States. In order to determine our accuracy, it is necessary 
that' we know what is actually in our sample segment.. The information 
requested for the segment that has been identified and forwarded’ to you 
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is essential for a successful- eval nation of the project. The enclosed 
color .prints have been obtained only over -the selected site in your 
county to support ground da,ta collection.. This is the only copy of the 
print and additional copies are not available for distribution to offices. 

B. Authority- . ■ - . 

The USDA LAC IE Project Manager has requested that the Agricultural 
Stabilization and Conservation Service provide this function and .they 
have accepted the assignment. You should have already gotten an authori- 
zation from your State office concerning this task. If you have not, you 
-should contact them at once. 

C. Requirements of the ASCS County Office 

You are being asked to do the following: - 

1. Review the set of instructions., 

2. Visit the segment- .location and identify the land uses, 
even if the segment, falls outside your county. 

3. Check over your work and return the completed inventory 
as soon as possible. , 

II. Data Collection Procedures 

A. Supplies . 

1. Color infrared print or prints. 

2. Mylar overlay. 

3. Topographic map -with segment location. 

4. Standard crop key- ' 

. 5. Crop stage development key. 

6. Forms to record plant height, ground cover observations 
and evaluation comments. 
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7. Return, post card and r- aim mailing tube -label. 

B. In some cases, all of the segment will not be covered by the 
photo. -Complete the survey for that portion outlined on the photo. 

C. Procedures- ; " • ' 4 -' 

1. You are. required to identify all fields within the segment 
boundaries- using v codes as indicated on the attached crop 
key (see attached TAOI-E segment classification). 

2. Use black -or red ball- point pen for all' coding directly on 

• . the mylar. • =• * • * 

3. The photos are provided as a base for field pattern and 

- references. ' • • 

a. All field identification should be based on actual 
ground conditions on .the' day that you visit the segment. 

b. If there- are any 'differences between the photo and the 

ground, then footnote each- field that is different and - 
explain on -evaluation form. . : ' 

c. Fields that are currently idle crop land should be 
marked as I/(code) and specifically identify the type 
of cover, i.e., ST = stubble; CC = winter seeded or 
volunteer growing crop;'RE>= disked or harrowed with 
residue; F .= fallow (clean tilled). 

4. Use the evaluation form for all comments on any unusual 
crop condition or practice (irregular,-, replanting, drought, 
etc.).' 
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5. If there are any crops in the segment for which there is no 
code, select an unused symbol and indicate its meaning on 
the evaluation -form. 

6. Assess the average wheat crop stages while completing the • 
segment inventory and enter it on the evaluation form upon 
completion. 

III. JSC Contact 

A. If there are any problems, contact the person listed below. 

B. Review procedures and crop key before going into the field and 
contact the Johnson Space Center if there are any questions. 

Bobby E. Spiers 

U.S. Department of Agriculture/ASCS 
NASA -.Johnson Space Center 
Large Area Crop Inventory Experiment 
Houston, TX 77058 

Phone: Commercial - A/C 713-483-4623 

FTS - 525-4623 

IV. Due Date and Mailing Procedures 

A. Upon receipt of data from the Johnson Space Center, complete the 
enclosed card and return it to JSC. 

B. Field information should be collected within 10 days after 
receipt of material by your office, if at all possible. 

1. Upon completion of field survey, fill out enclosed forms 
and return with photos. 

2. Return all evaluation forms and photos in a mailing tube, 
using the provided return label. 

C. Thank you for your cooperation and effort in assisting LAC IE 
in this vital area of the experiment. 
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SAMPLE SEGMENT 


With Photo and Mylar 
(Not to Scale) 
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Sample : 11/ A 


These were winter wheat fields hut are currently 
being plowed under due to" the effects of drought. 
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STANDARD CROP KEY 


KEY / 

W 

SW 

G 

H 

A . 

P 

C 

SF 

SU ■ 

SG 

SR 

SY 
SB • 

FX 

T 

R 

B 

X 

BN 

0 

(Crop)/H 

(Crop)/A 


CROP TYPE ' '■ 

Winter Wheat 
Spring Wheat 

Grass (not cut for hay and no fence ) v 
Hay (any visible signs of hay activities) 

A1 f al fa 
Pasture 
Corn 

Saffl ower 

Sunflower 

Sudan Grass 

Sorghum 

Soybeans 
Sugar' Beets 
Flax 

Trees 

Rye . 

Bari ey 

Homestead - nonag, lakes, ponds, etc. 

Beans 

Oats 

Crop has been harvested, 1977 crop 

Crop has been abandoned^ footnote and explain, wii crop 


1/ (Code) 


Idle crop land/ST = stubble; cc = growing cover crop; 
RE = residue; F = fallow (clean tilled) 


1. Do not use the code W or SW for any crop other than wheat. 

2. If there are crops in segment for which there is no code, select an 
unused symbol and indicate its use on the evaluation form. 


3. Use standard key for all identification. 

4. Use ball point pen for all coding on mylar. 


D-9 



CROP STAGE KEY 

i 

:ROP STAGE KEY STAGE DESCRIPTION 

. 1.0 Planted Seed was put in the ground. 

2.0 Emerged - • When one leaf per plant is visible.. 

3.0 Jointed - Defined as when the first node of the stem 

is visible. 

4.0. - Heading' Defined as-the stage when the base of the 

rachis (or head) reached the same height 
as the ligule (or base of the shot leaf). 

5.0 Soft Dough. At this stage the crop is starting to turn 

color. The kerhals can be easily deformed 

* when. pressed between the fingers, but no 

• “milk” or liquid should exude under such ' 
pressure. 

Hard Dough The kernals readily part from the head. 

The grain is firm and though it may be 

• dented by pressure of the thumbnail, it is not 
easily crushed. The characteristic color of 
the grain has become more distinct. The 
leaves are brown, dry, and shrunken. Wheat in 
this stage may be swathed in some areas. 

7.0 ’ Harvested or Straw is brittle and dull yellow at this 

Harvestable stage. The grain (if not., harvested yet) ■ 

. is hard and breaks into fragments when 
crushed. 


• 6.0 
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EVALUATION FORM 

Segment No.: ~ County: State: 

Name: . Date: 


Man-Hours Required to Complete Survey: Mileage: 

I. Based on your assessment o'f the development of wheat in the segment 1 
while completing the survey, what is the average wheat stage for .the 
segment?- See attached Crop- Stage. Key, Is- the crop development this 
year in the segment -normal, ahead, or behind as compared to previous 
years? ' Explain. - Enter Crop. Stage: ' ■ 


II. Conments, -footnotes, and additional crop key. used: 


i 

III.- Comments on the effects of drought and/or winterkill.: 


IV. Comments and recommendations for improving these .procedures for future 
surveys: 
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Plant Height 
and 

Percent Ground Cover Observations 


Segment Number^ 
Observation Date" 


1) During and/or after yoursegment inventory complete this form -and -obtain plant height 
and cover in 15 segment wheat fields. • . . * ” " 


. 2) These fields should represent a range from below average to above average stands of 
Wheat for the sample segment only. _ -* ' 

3) Select five fields in each category below average , average , and above average based 
on your- judgment of the overall average condition -of wheat in the sample segment -only . 

4) Plant height and percent' ground .’cover should'be measured at the same location within 
the field and ''should represent the average condition of the field. 

5) Numbers should be assigned to fields according to the table below, fields 1-5 below 
average, 6-10 average,, 11-15, above average. Fields can be selected and data collected 
in any order. 

. 6) Plant height should be measured to the nearest "inch. 

7) Ground cover should be measured according to the following codes: 1(0-19%), 
2(20-39%), 3(40-59%), 4(60-79%), 5(80-100%). This measurement is taken by looking 
straight down at the ground and estimating the percent of soil that is covered by the 
crop, and v assigning the appropriate code to the field. 


Data Collection Requirements : : 

•You are required to select fields and determine which of three -categories the field 
should be assigned to. Select the next -avail able field number in that category. Enter 
the field from the most accessible corner or turn row. Proceed into the field at least 
20 paces to a spot that you feel represents -the average condition of the field and 
proceed to obtain plant height and percent ground cover. Write the field number, on 
the overlay at the exact location that the measurements were obtained and circle the 
number. Continue with the inventory or field selection. 
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FORMS USED IN TAKING INVENTORIES, 
. OF LACIE INTENSIVE TEST SITES 

The following 
LACIE ITS's: 

forms are used to record ground observations in -the 

Figure 

Form 

E-l 

Sample Ground Truth Inventory Form 

E-2 

- -Sample -Ground Truth Periodic Observation 
Form 

E-3 

Sample form for Ground Truth Data . Collection 
System Rainfall Measurements 

E-4 ' 

Sample Yield Form 
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1N ' M 


ttinuWihisiih 

200 BARLEY (GEN) 

201 LACKER 

202 VANGUARD 
20) bCNAN2A . 

206 UltKSVN 
2 os rhinus 

706 SHAkiifcT 

707 pimhine 
70,1 6E17CS . 

20/ SI6C10E 
210 FERGUS 


MAP REftflENCfe 
0 Vf- FIELD 


ACREAGE 


211 PARAGON 

212 Hill T A 

21 A CI-NGUCS7 
21 A WISCONSIN 
215 HCClOR 

213 Mi CAM 
217 MORAVIAN 
200 UTS (GEN) 
501 FUS5LLL 

302 BASIN 

303 KILSFV 
306 RCUNEY 
305 LODI 

LAND USE 
CROC COOS 


GROUND TRUTH INVENTORY FOIIM 


LAND USE CROP CGGES 


306 KINSEY 
30? M Art MON 
305 MISSION 

600 wlrit C« WHEAT (GEN) 
AO l OAIX HARD NORTHERN 

60 2 Scour 

603 PARRER 
606 eagle 

605 APACHE 

606 1MUKPH 

607 SAIANTA 
600 CCNIURK 
609 0 1 SCN 

IRRIGATED (FERTILIZED 


610 W.NStft 

611 MCCALL 

612 SKiVOR 

61 i GUNIS 

616 NU0A1NES 
615 ORoCMAMP 

6 1 o mopo 

617 oucr 
610 OKI ODER 

619 AA7HUA 

620 TASCOSA 

62 1 STURDY 
622 CCNCMO 


'•// // 

DATA RECORDED BeTWjJOj ?/(£ f ,]\ 


623 CAl’RCCK 

625 Cagle scour 

623 TRAPPER 

626 CMLYLNNE 

627 MllitUA 
‘ ‘ ' WIHCKA 

GR AS S CS/P A STURE BTC. 
otiii.h crups TgenT 
rapeseeo 

RYE 

MUSTARD 

606 FLAX (.GEN) 

■605 NOHALT-A FLAX 


606 ft LOW Olio FLAX 

607 CORN 

610 OttLKHHCAT 

611 SUHfLOWCRS 

612 DRY OEANS 

613 LENTILS 
616 URY PEAS 

615 SUGAR BEETS 

616 GRAIN SORGHUM 
*17 SOYCI ANS 

610 DOI TON 
700 SUMHLK FALLOW 
1)00 Nllfl-AGRICULIURE 
900 UNKNOWN CROPS 
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MONTH day year 
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Figure E-I.— Sample Ground Truth Inventory Form. 




GROUND TRUTH PERIODIC OBSERVATION FORM 
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TEST SITE § 25 (SHELOY, INDIANA) 
OBSERVATION 01 


LAND USE CODES 

100-5PRING WHEAT 
200 8ARLEY 
300-OATS 

400-WtNTEP WHEAT 
500-GP AS SF S/PAS TURE 

600- 0 T HE R CROPS 

60 1- R APE SEED 

602- RYC 
604 -PL AX 
607-CORN 
617-SOYBEANS 
616-COTTGM 
700-5UMMER FALLOW 
900-UNKNOWN CROPS 


GROWTH STAGES GROUND. 

COVER U) 

UI-NOT PLANTED 

02- PLANTE0 NO EMERGENCE 1-0- 19 

03- EMERGENCE 2-20- 39 

04- TMLFR I NG» PREBOOT* 3-40- 59 

PKERUO 4-60- 79 

05- OOOTLO OR BUDDEO 5-UO-lOO 

06- BEGINNING TO HEAD 

OR' FLOWER 

07- FUU Y HEADED OR 

F LOWERED 

08- 0EGINNING TO RIPEN 

09- RIPE MATURE 

10- HARV6STF0 

11- DOES NOT APPLY 


LANOSAT PASS DATE 
OBSERVATION DATE 


"SSfTV 


DAY 


SURFACE MOISTURE 
CONDIT IONS 

1- DKV 

2- OAHP 
3 WET 

4-STANDING WATER 


WEED GROWTH 

t-NEGL IGltiLE 

2- SLIGHT 

3- KfiDERATE 

4 -H6AV/ *■ 


FJELO OPERATIONS 

0 1-BAR E GROUND 

02- DARE DISKEIVCULTIVATED 

03- DARE PL9WE0 

04- BAR ESECDED 

05- STAND 1NG STUBBLE 

06 S TUDULfc DISKEO/CULTIVATED 
07-STUBBLE PLOWED 
DO- STUBBLE SEEDED 

09 - BURNED 

10- GRAZbO „ 

1 1- W 1 NDROWED OR SWATHED 


RAINFALL SINGE LAST QB5^&VAT?Ufl 


/T6 

/76 


.IN* 


GBOWTH/YI ELD 
OCTB ACTANTS 


STAND 

QUALITY 


12- MOWED OR C0H8INE0 

13 - STACKED OR BALED 

14- 0 


-OTHER 


01 -SALINITY i-POOR 

02- INSECTS 2-BE10W 

03- OJSEA5E AVERAGE 

04 - OR OUGHT 3 -AVERAGE 

05- Hl) 1 STUHE 4-AdOVh _ 

■ Od-HINO AVERAGE 

07 HAIL 5 EXCELLENT 

Od-FRIiST 6 -DOES NOT 

09 -BIRDS APPLY 

10- POT HOLFS 13-WINTERKILL 

1 1- UNf Vr N STAND 14-LODGING 

12- WEEDS 15-OTHER, 


FIELO 

NO. 

ACREAGE 

LAND USE 
CODE 

GROWTH GROUND 

. stage cover 

1 (CIRCLE (CIRCLE 
ONE) ONE) 

PLANT 
HEIGHT • 
(INCHES) 

SURFACE 

MOISTURE 

(CIRCLE 

ONE) 

WEED 

GROWTH 

(CIRCLE 

0N6I 

f 1 eld ■ 

OPERATIONS 
(CIRCLE 
ONE ) 

GROWTM/YIELD 
DETRACTANTS 
(CIRCLE ONE) 

STAND DUALITY 
RATING 
(CIRCLE ONE) 
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Figure E-2.~ Sample Ground Truth Periodic Observation' Form. 
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Figure E-3.— Sample form for Ground Truth Data Collection System Rainfall Measurements. 




Figure E-4.— Sample Yield Form 
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GROWTH STAGE OATES FOR 10% DEVELOPMENT 
MONTH AND DAY OF MONTH 


Check one: 

□ Spring Wheat 

□ Winter Wheat 

Crop Year 


ITS SEG. NO. 

CRD ii 

PLANTING 

DATE 1 

EMERGENCE 
DATE 1 ' 

JOINTING 

DATE 2 

HEADING 

DATE 2 

SOFT 

DOUGH 

DATE 3 

RIPE' 

DATE 1 * 

HARVEST 

DATE 5 

! 

! 

• 

I 

| 

1 

-i 

! 

1 


1 

! 


I 




1 Date at which 1035 of fields in CRD were planted or emerged, respectively. 

2 Date at which 102 of fields in CRD had - begun to joint or head, respectively. . 

3 Date at which 102 of fields In CRD had begun to enter soft dough stage (turning color to greenish-yellow to yellow). 

" Date at which 102 of fields in CRD are ripe (hard dough) -stage or when they were swathed, (Indicated swathed If 

applicable). 

1 Date at which 102 of fields in CRD have been harvested either as standing grain or out of swath. 


GROWTH STAGE DATES FOR 50% DEVELOPMENT 
MONTH AND DAY OF MONTH 


Check one: 

Q SprJng Wheat 
Q Winter Wheat 
Crop Year . 


ITS SEG. NO. 

CRD 0 

PLANTING 
' DATE 1 

EMERGENCE 
DATE 1 • 

JOINTING 

DATE 2 

HEADING 

.DATE 2 

SOFT 
DOUGH 
DATE 3 ■ 

'RIPE 

DATE 1 * 

■ HARVEST 
DATE 5 


■ 

. 




• 




1 Date at which 505! of fields in CRD were planted or emerged, respectively. 

* Date at which 505! of fields in CRD had - begun to joint or head, respectively. 

3 Date at which 5055 of fields in CRD had begun to enter soft dough stage (turning color to greenish-yellow to yellow). 

k Date at which 50% of fields in CRD are ripe (hard dough) stage or when they were swathed, (Indicated swathed if 

•applicable). 

. 5 Date at which 50% of fields in CRD have been harvested either as standing grain or out of swath. 



GROWTH STAGE DATES FOR 90% DEVELOPMENT 


Check one: 

0 Spring Wheat 
0 Winter Wheat 


MONTH AND OAY OF- MONT H Crop Year 


ITS SEG. NO. ‘ 

CRO tf 

PLANTING . 
'DATE 1 

EMERGENCE 
' DATE* 

• JOINTING ‘ 
DATE 2 

HEADING 

DATE 2 

SOFT 

DOUGH 

DATE 1 

RIPE 

DATE 1 * 

HARVEST 

DATE 5 

' V . 

1 

• 



• 






1 Date at which 90 % of fields In CRD were planted or emerged, respectively. 

2 Date at which 9035 of fields in CRD had begun to joint or head, respectively. 

5 - Date at which 90£ of fields In CRD had begun to enter soft dough stage {turning color *to greenish-yellow to yellow). 

" Date at which 9035 of fields in CRD are ripe (hard dough) stage or when they were swathed. (Indicated swathed if 

appl icable). 


5 Date at which 903! of fields in CRD have been harvested either as standing grain or out of swath. 
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APPENDIX ' G 


PHASE III CAMS OPERATIONAL PROCEDURES FLOW DIAGRAMS 



A PHASE I PROCEDURES (October »76 - M*rch 28 1977) 


FLOW DIAGRAMS OF CAMS PHASE OPERATING PROCEDURES 



8 SMALL FELDS PROCEDURE - LACtE 5A (Much 28. 1977 - June 1977) 
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) IMAGE 100 PROCEDURE 1 (June 15 1977 Until End ci Phase B) 



ORIGINAL page is 
PI KOOR QUALITY 


G-l 























^CEDING PAGE BLANK NOT FlLfft£& 


APPENDIX H 

PHASE III AA PIXEL-LEVEL 
DATA PROCESSING SYSTEM 



APPENDIX H 

PHASE III.AA PIXEL-LEVEL 
DATA PROCESSING SYSTEM 

In Phase III, AA will conduct pixel-level investigations for the 
first time. These are designed to monitor the performance of 
the various components of the LACIE system (analysts, clustering 
algorithm, and classification algorithm) and to investigate the 
sources of error in these elements. These investigations are 
described in section 6. 2. 5. 2. This appendix describes the data 
processing system that has been designed to support the pixel- 
level investigations. In addition, this system automatically 
produces some data (such as segment wheat proportions) that are 
used in segment-level, investigations. 

The system involves the processing of .ground-truth, data, CAMS 
classification maps, CAMS cluster maps, and CAMS analyst- 
interpreter- selected dot labels. A flow diagram of this AA data 
processing system is presented in figure H-l. The Phase III AA 
processing described in this figure includes four major stepsj. 

1. CAMS DTRM tape preprocessing 

2. AA ground-truth preprocessing 

3. CAMS analyst dot label preprocessing 

4. AA classification/label accuracy processing 

The first three - steps are necessary to prepare for the fourth 
one. This last processing step. is done using the computer pro- 
grams SPATL and SPECTL on the PDP 11/45. 

Since only about 10 percent of the segments on the DTRM tapes are 
U.S. blind sites, a preprocessing' program is required. This 


H-l 



program will strip the blind site data from the DTRM tape and put 
it onto another tape containing only blind site data. 

The DTRM tape contains all~of the classification maps and condi- 
tional and unconditional cluster maps for all segments processed 
by CAMS, beginning with the segments processed using the small 
fields procedures . ' Since the identity of the blind' sites is to 
remain unknown to : 'the ’GAMS analysts, AA will provide' blind site 
identification cards' 1 as input’- for the DTRM tape- searching program. 
When data" for a - ' blind site 'are located on the DTRM tape, they will - 
be copied onto the AA blind -site class/cluster map tape (the 
first step in the processing' task) .' 

The second step is the conversion of ground-truthmaps into 
ground-truth tapes -suitable for subsequent processing. This 
initially involves the use of the Dell Foster equipment which 
produces field vertices cards. These cards are converted to a 
f tape, by the program BTAPE 'to produce 'a tape with a format similar 
to the tape created by the Bendix 100 Interactive Drafting System. 
This format contains the ; x and y coordinates of the vertices of 
the agricultural fields from 5 the aerial ’photographs of the blind 
sites. The" resultant 'product is a inine-track’ computer-compatible 
tape (CCT) containing a field identification. -and field 
coordinates. 

The Dell Foster equipment will be ’used for early season process- 

r 

ing until the Bendix 100 software package-, for the Interactive 
Drafting System becomes available and is checked out. The sched- 
ule for the beginning, of .ground-truth map processing using >the 
Bendix 100 is, June 15., 1977. .The Bendix- 100 tapes will be proc- 
essed by the programs PHASE! and -PHASE 2 to- produce' -a Universal 
format ground- truth tape containing AA crop labels at the sub- 
pixel .level, ■ registered to the Landsat, coordinates. This AA 
ground- truth tape will be coded > as per;- the .AA codes presented in 
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tablet 6-2. The, ground-truth , tape is. the .product of the. second 
step in the AA processing system,, ' „. ,. 

The third step requires SF3 to acquire the SF4 CAMS .pixel-label.:, 
sheets which contain the analyst dot labels and classification . 
dot labels prepared during the Phase III operational processing. 
These sheets were initially used in operations when the small 
fields procedures were implemented and will be used throughout 
the remainder of Phase III. From these sheets, SF3 will keypunch 
dot label cards to be used as input to the AA accuracy calcula- 
tions (step 4) . The AI dot label card deck or tape is the out- 
put of step 3. As Procedure 1 implementation progresses, the 
CAMS/CAS interface tape may be used to obtain the analyst- 
interpreter dot labels instead of the analyst- interpreter dot 
label cards. 

The AA classification/labeling processing element utilizes pro- 
grams SPATL and SPECTL to compare the class/cluster map tape, 
the ground-truth tape, and the analyst-interpreter dot labels. 

This comparison is done. at the pixel and segment levels, which 
generate data for subsequent AA evaluation. For additional 
information regarding this processing step, refer to technical 
memorandum LEC-10620 (ref. 4), entitled; "Requirements Document 
For Phase III LACI-E Accuracy Assessment." 

Later in Phase III, an AA data file will be used as indicated 
in figure H— 1. The PDP 11/45 will be. used to store the three 
basic inputs on the disk (c las s/cluster maps, ground truth, and 
analyst dot labels) . The fourth-element programs will use 
this disk as a data base. 

Figure H-l indicates the processing responsibilities of SF3 and 
SF12. The responsibilities of SF3 are contained within the large 
I dashed lines, with the remaining elements being the responsibility 
of SF12 . 
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in order to assure that the results from the programs SPECTL and 
SPATL are correct, some of the data should be processed inter- 
actively on the |lMAGE 100 system. It is anticipated that about 
one segment per week or about 5 percent of the blind sites should 
be processed in this manner. About 4 hours of (IMAGE 100 time 
per segment will be required. 
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Figure H-l.~ Phase III Accuracy Assessment data processing system flow diagram. 
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APPENDIX I 


AA 

AA-01 

ACC 

Agromet 

Ai 

I Biowindow 
| biophase 


Biostage 


ABBREVIATIONS , ACRONYMS, AND SYMBOLS 


j Accuracy Assessment. 

LACIE Phase II AA Report for February 1976. 
adjustable crop calendar, 
agricultural/meteorological . 
analyst-interpreter . 

or biological window — a Landsat data acquisi- 

tion period that is related to the biostages 
of wheat development. The LACIE approach is 
based on the judgment that wheat can be sepa- 
rated adequately from other crops by analysis 
.of up to four acquisitions of Landsat data 
during the growing season. The biowindow may 
be updated if there is a significant lag or 
advancement in the current crop calendar. 

The sequence chosen includes acquisitions 
during the following biowindows: 

1. Crop establishment — from 50 percent 
tillering to 50 percent jointing (bio- 
stage 2.3 to 3.0). 

2. . Green — from 50 percent jointing to 

.50 percent heading (biostage 3.1 to 4.0). 

3. Heading — from 50 percent heading to 
50 percent soft dough (biostage 4.1 to 
5.0) . 

4 . Mature — from 50 percent soft dough to 

50 percent harvest (biostage 5.1 to 6.0). 

biological stage — the specific stage of 
development of a crop which can be recognized 
by a major change in plant structure; i.e., 
emergence after germination, jointing, heading, 
soft dough, ripening,, and harvest, which are 
• represented by integers on the Robertson 
l Biometeorological Time Scale. 
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Blind site 


a LAC IE sample segment that is a part of the 
LACIE operational random sample set of seg- 
'ments used in the LACIE- aggregations. The 
blind sites are not identified to analyst^ j 
in order to ensure normal processing of the 
blind site segments. However, in Canada, 
where only the blind site segments are being 
processed, the analyst obviously knows their 
location. 

BMTS Biometeorological Time Scale. 


CAMS 

CAR 

CAS 

CCEA 


Classification 


Classification 

error 

CMR 


Classification and Mensuration Subsystem. 

CAS ^Annual Report 

Crop- Assessment Subsystem. 

Center for Climatic and Environmental | 

Assessment an organization of the National 
Oceanic and Atmospheric Administration, 
Columbia, Missouri. 

in. computer-aided analysis of remotely sensed 
data, the process of assigning data points to 
specified classes by a testing process in 
which the -'Spectralj properties of each unknown 
data point’ are compared with spectral proper- 
ties typical of the subject being classified. 

a measure of the degree to which the LACIE 
CAMS overestimates or underestimates the wheat 
acreage in one or more LACIE samples. 

CAS. Monthly Report. 


COM 


classification on microfilm. 


CRD Crop Reporting District — a geographical area 

used by the U.S. Department of Agriculture 
for the collection and reporting of agricul- 
tural information; each district consists of 
several counties. 

Crop calendar a calendar depicting the-, bio stages of the 

major crop types within a specified region 
during a calendar , year . - 


1-2 



Crop calendar 
adjustment' 

CUR 

CV 

DAPTS 

DO/DU 

DOY 

First-order error 
source evaluations 

Group 2 segment 

Group 3 segment 

IE 

IMR 

ISRRS 

ITS 

JSC 
LAC IE 


an' adjustment made to. the normal crop calen- 
dar on the basis of current meteorological 
data . ‘ ■ 

CAS Unscheduled ' Report . 

coefficient of variation (standard deviation 
divided by the mean) . 

Data Acquisition, Preprocessing, and Trans- 
mission Subsystem. 

’designated other/designated unidentifiable, 
day of the year ' 

evaluations to identify the major components 
of error in LACIE Phase III production esti- 
mates; these components can be statistically 
estimated with techniques described -in 
section 6.2.4 of the report. 

LACIE segment in a county that historically 
produces small quantities of wheat/small 
grains; samples are allocated with probability 
proportional to size. 

LACIE segment in a county that historically 
produces very small quantities of wheat/small 
grains; estimates are based on the changes in 
' acreage of group 1 and 2 segments from year 
to year. 

Information Evaluation. 

IE Monthly Report. 

Information, Storage, Retrieval, and Reformat- 
ting Subsystem 

intensive test site - a LACIE segment in the 
United States or Canada over which detailed 
crop 'information is collected by using 
ground and airborne observation equipment; 
these ITS segments are a separate set of 
segments' from the operation LACIE segments 
used in the aggregations. 

Lyndon B. Johnson Space Center of NASA, 
Houston, Texas.' 

Large Area Crop Inventory Experiment. 
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Land sat 


LEC 

LPDL 

LPP 

MSS 

Multitemporal 

analysis 

NASA 

NOAA 

90-90 criterion 

t 

PCC 

PFC 

PPS 

RD 

RTEB 

Sample segment 
Sampling error 


■Land Satellite — formerly called ERTS (Earth. 
Resources Technology Satellite) ; operates in 
a circular. Sun- synchronous , near-polar orbit 
of the Earth at an altitude of . approximately 
915 kilometers; orbits the Earth about 14 times 
a day and views the same scene approximately 
every 18 days. 

Lockheed Electronics Company, Inc. 

LACIE Physical Data Library. 

LAC IE. Performance Predictor. 

Multispectral Scanner System or multispectral 
scanner — the remote sensing instrument on 
Landsat that measures reflected sunlight on 
various, spectral bands or wavelengths.' 

analysis of data sets over the same area 
acquired at different times during the grow- 
ing season. 

National Aeronautics and Space Administration. 

National Oceanic and Atmospheric Administration. 

criterion that the LACIE U.S.- Great Plains - 
production estimate is 90 percent accurate, 
at harvest, 90 percent of the time (in com- 
parison with the true value.) . 

probability of correct classification. 

production film converter. 

probability proportional to size. 

relative difference. 

Research, Test, and Evaluation Branch. 

a 5- by 6-nautical-mile area selected by 
stratified random sampling; information is 
recorded by the MSS and transformed into 
computer-compatible tapes and film products. 

a measure of the- degree to which the esti- 
mated wheat acreage in the LACIE sample seg- 
ments does not represent the wheat acreage 
contained in’ the survey region being sampled. 



Second-order error /evaluations to further qualitatively break 

source /evaluations down the identified first-order components 

into factors that can be related to causal 
elements in LAC IE methods and procedures. 

U.S. Department of Agriculture. 

USDA Agricultural Stabilization and Conserva- 
tion Service . 

USDA/FAS USDA Foreign Agricultural Service. 

USDA/SRS USDA Statistical Reporting Service. 

U.S. Great Plains an area encompassing the nine states of 

Colorado, Kansas, Minnesota, Montana, Nebraska, 
North and South Dakota, Oklahoma, and Texas; 
it is divided geographically into (1) the U.S. 
southern Great Plains, which includes Colorado, 
Kansas, Nebraska, Oklahoma, and Texas, and 
(2) the U.S. northern Great Plains, which 
includes Minnesota, Montana, and North and 
South Dakota. 

WMO World Meteorological Organization. 

YES Yield Estimation Subsystem. 


SYMBOLS : 

A 

P wheat/ small-grain proportion estimate. 

P w proportion of wheat harvested. 

a standard deviation. 

A 

CV (W) CV for production estimate. 

Pgrj, proportion of wheat/small grains based on 

identification of each field in the blind 
site or ITS by USDA/ASCS personnel. 


USDA 

USDA/ASCS 
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